[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251214 02:57:26.651763 19981 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.131.126:44625
I20251214 02:57:26.653301 19981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 02:57:26.654330 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:26.670488 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:26.670481 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:26.671866 19981 server_base.cc:1047] running on GCE node
W20251214 02:57:26.670469 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:27.757360 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:27.757611 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:27.757767 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681047757752 us; error 0 us; skew 500 ppm
I20251214 02:57:27.758528 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:27.763612 19981 webserver.cc:492] Webserver started at http://127.19.131.126:40973/ using document root <none> and password file <none>
I20251214 02:57:27.764425 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:27.764609 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:27.764990 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:27.770709 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/master-0-root/instance:
uuid: "c1368a111fd448f6a306ed9c107462fa"
format_stamp: "Formatted at 2025-12-14 02:57:27 on dist-test-slave-8pjb"
I20251214 02:57:27.778203 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20251214 02:57:27.783169 19996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:27.784265 19981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20251214 02:57:27.784560 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/master-0-root
uuid: "c1368a111fd448f6a306ed9c107462fa"
format_stamp: "Formatted at 2025-12-14 02:57:27 on dist-test-slave-8pjb"
I20251214 02:57:27.784857 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:27.830269 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:27.831883 19981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 02:57:27.832268 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:27.903569 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.126:44625
I20251214 02:57:27.903658 20047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.126:44625 every 8 connection(s)
I20251214 02:57:27.911208 20048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:27.937958 20048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: Bootstrap starting.
I20251214 02:57:27.943754 20048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:27.945585 20048 log.cc:826] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: Log is configured to *not* fsync() on all Append() calls
I20251214 02:57:27.950105 20048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: No bootstrap required, opened a new log
I20251214 02:57:27.971122 20048 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER }
I20251214 02:57:27.971664 20048 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:27.971863 20048 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1368a111fd448f6a306ed9c107462fa, State: Initialized, Role: FOLLOWER
I20251214 02:57:27.972568 20048 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER }
I20251214 02:57:27.973019 20048 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 02:57:27.973246 20048 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 02:57:27.973532 20048 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:27.978714 20048 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER }
I20251214 02:57:27.979666 20048 leader_election.cc:304] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1368a111fd448f6a306ed9c107462fa; no voters:
I20251214 02:57:27.981341 20048 leader_election.cc:290] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 02:57:27.981629 20051 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:27.983701 20051 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 1 LEADER]: Becoming Leader. State: Replica: c1368a111fd448f6a306ed9c107462fa, State: Running, Role: LEADER
I20251214 02:57:27.984633 20051 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER }
I20251214 02:57:27.985772 20048 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [sys.catalog]: configured and running, proceeding with master startup.
I20251214 02:57:27.995010 20052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1368a111fd448f6a306ed9c107462fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER } }
I20251214 02:57:27.995738 20052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [sys.catalog]: This master's current role is: LEADER
I20251214 02:57:27.996573 20053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1368a111fd448f6a306ed9c107462fa. Latest consensus state: current_term: 1 leader_uuid: "c1368a111fd448f6a306ed9c107462fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1368a111fd448f6a306ed9c107462fa" member_type: VOTER } }
I20251214 02:57:27.997246 20053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [sys.catalog]: This master's current role is: LEADER
I20251214 02:57:28.000181 20060 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 02:57:28.010483 20060 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 02:57:28.012283 19981 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251214 02:57:28.025738 20060 catalog_manager.cc:1357] Generated new cluster ID: b99508c1c1fc45ad8e08168506f03989
I20251214 02:57:28.026057 20060 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 02:57:28.049836 20060 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 02:57:28.052011 20060 catalog_manager.cc:1514] Loading token signing keys...
I20251214 02:57:28.067073 20060 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: Generated new TSK 0
I20251214 02:57:28.068200 20060 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 02:57:28.078682 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:28.086440 20070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:28.087764 20071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.092123 19981 server_base.cc:1047] running on GCE node
W20251214 02:57:28.092504 20073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.093317 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:28.093514 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:28.093678 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681048093662 us; error 0 us; skew 500 ppm
I20251214 02:57:28.094247 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:28.098240 19981 webserver.cc:492] Webserver started at http://127.19.131.65:39109/ using document root <none> and password file <none>
I20251214 02:57:28.098860 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:28.099035 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:28.099301 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:28.100366 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-0-root/instance:
uuid: "9269ad98b10f4d61a2d9a943a117684d"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.105871 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251214 02:57:28.110081 20078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.110919 19981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251214 02:57:28.111265 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
uuid: "9269ad98b10f4d61a2d9a943a117684d"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.111662 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:28.133356 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:28.135106 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:28.138083 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:28.143432 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:28.143733 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.144032 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:28.144253 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.235045 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.65:33917
I20251214 02:57:28.235101 20140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.65:33917 every 8 connection(s)
I20251214 02:57:28.246001 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:28.254647 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:28.258464 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.260702 19981 server_base.cc:1047] running on GCE node
W20251214 02:57:28.261445 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.262485 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:28.262856 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:28.263111 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681048263095 us; error 0 us; skew 500 ppm
I20251214 02:57:28.263890 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:28.265237 20141 heartbeater.cc:344] Connected to a master server at 127.19.131.126:44625
I20251214 02:57:28.265628 20141 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:28.266469 20141 heartbeater.cc:507] Master 127.19.131.126:44625 requested a full tablet report, sending...
I20251214 02:57:28.269253 20013 ts_manager.cc:194] Registered new tserver with Master: 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917)
I20251214 02:57:28.271107 19981 webserver.cc:492] Webserver started at http://127.19.131.66:39883/ using document root <none> and password file <none>
I20251214 02:57:28.271533 20013 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39996
I20251214 02:57:28.271785 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:28.272058 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:28.272390 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:28.273950 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-1-root/instance:
uuid: "974a58ccac3b4728a98b46838595d05e"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.280448 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251214 02:57:28.284524 20153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.285310 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 02:57:28.285630 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
uuid: "974a58ccac3b4728a98b46838595d05e"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.285964 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:28.304010 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:28.305625 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:28.308104 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:28.311693 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:28.311946 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.312222 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:28.312424 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.400171 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.66:38649
I20251214 02:57:28.400256 20215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.66:38649 every 8 connection(s)
I20251214 02:57:28.408618 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:28.417515 20219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.418143 20216 heartbeater.cc:344] Connected to a master server at 127.19.131.126:44625
I20251214 02:57:28.418745 20216 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:28.419785 20216 heartbeater.cc:507] Master 127.19.131.126:44625 requested a full tablet report, sending...
W20251214 02:57:28.420320 20220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.422143 20013 ts_manager.cc:194] Registered new tserver with Master: 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:28.424024 20013 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40002
W20251214 02:57:28.428591 20222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:28.429054 19981 server_base.cc:1047] running on GCE node
I20251214 02:57:28.429888 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:28.430070 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:28.430225 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681048430209 us; error 0 us; skew 500 ppm
I20251214 02:57:28.430759 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:28.433120 19981 webserver.cc:492] Webserver started at http://127.19.131.67:43649/ using document root <none> and password file <none>
I20251214 02:57:28.433565 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:28.433753 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:28.433987 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:28.434988 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-2-root/instance:
uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.439370 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251214 02:57:28.442710 20227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.443465 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251214 02:57:28.443723 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
format_stamp: "Formatted at 2025-12-14 02:57:28 on dist-test-slave-8pjb"
I20251214 02:57:28.443980 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1765681046489641-19981-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:28.462026 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:28.463294 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:28.465153 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:28.468075 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:28.468267 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.468489 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:28.468645 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:28.556275 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.67:35957
I20251214 02:57:28.556483 20289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.67:35957 every 8 connection(s)
I20251214 02:57:28.573520 20290 heartbeater.cc:344] Connected to a master server at 127.19.131.126:44625
I20251214 02:57:28.573925 20290 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:28.574551 20290 heartbeater.cc:507] Master 127.19.131.126:44625 requested a full tablet report, sending...
I20251214 02:57:28.576431 20013 ts_manager.cc:194] Registered new tserver with Master: 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:28.577252 19981 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014179716s
I20251214 02:57:28.577926 20013 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40018
I20251214 02:57:28.580780 19981 test_util.cc:276] Using random seed: -1481641322
I20251214 02:57:28.624547 20013 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40024:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 02:57:28.627990 20013 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 02:57:28.721591 20106 tablet_service.cc:1505] Processing CreateTablet for tablet 770832072b40455b9a76ba3e907fb517 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251214 02:57:28.721767 20100 tablet_service.cc:1505] Processing CreateTablet for tablet 99fd1f7626cf406db2c278a08948befb (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251214 02:57:28.729638 20100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd1f7626cf406db2c278a08948befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.729336 20102 tablet_service.cc:1505] Processing CreateTablet for tablet c55d8a42b8694ebf9b76fbb571a9814d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251214 02:57:28.728397 20104 tablet_service.cc:1505] Processing CreateTablet for tablet 26f95721f9c4489892d51ed0aa794585 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251214 02:57:28.731033 20105 tablet_service.cc:1505] Processing CreateTablet for tablet c411c9b2da7e4e2b83d854f004f13bfa (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251214 02:57:28.724839 20101 tablet_service.cc:1505] Processing CreateTablet for tablet fcdae6c6dc2445d591d0bbefed18e4b9 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251214 02:57:28.727442 20103 tablet_service.cc:1505] Processing CreateTablet for tablet cb9ad90d00ed4e249f54ebbe0b02e22e (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251214 02:57:28.732697 20106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770832072b40455b9a76ba3e907fb517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.735572 20103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9ad90d00ed4e249f54ebbe0b02e22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.737730 20105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c411c9b2da7e4e2b83d854f004f13bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.741446 20099 tablet_service.cc:1505] Processing CreateTablet for tablet bea937187ce744e99c7e48d4afc3469d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251214 02:57:28.741034 20101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcdae6c6dc2445d591d0bbefed18e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.743256 20099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea937187ce744e99c7e48d4afc3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.744539 20102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55d8a42b8694ebf9b76fbb571a9814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.747334 20098 tablet_service.cc:1505] Processing CreateTablet for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251214 02:57:28.748106 20104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f95721f9c4489892d51ed0aa794585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.750259 20098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.755323 20255 tablet_service.cc:1505] Processing CreateTablet for tablet 770832072b40455b9a76ba3e907fb517 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251214 02:57:28.760104 20254 tablet_service.cc:1505] Processing CreateTablet for tablet c411c9b2da7e4e2b83d854f004f13bfa (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251214 02:57:28.761404 20254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c411c9b2da7e4e2b83d854f004f13bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.762876 20253 tablet_service.cc:1505] Processing CreateTablet for tablet 26f95721f9c4489892d51ed0aa794585 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251214 02:57:28.768080 20251 tablet_service.cc:1505] Processing CreateTablet for tablet c55d8a42b8694ebf9b76fbb571a9814d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251214 02:57:28.771713 20255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770832072b40455b9a76ba3e907fb517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.775471 20249 tablet_service.cc:1505] Processing CreateTablet for tablet 99fd1f7626cf406db2c278a08948befb (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251214 02:57:28.783496 20251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55d8a42b8694ebf9b76fbb571a9814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.774266 20252 tablet_service.cc:1505] Processing CreateTablet for tablet cb9ad90d00ed4e249f54ebbe0b02e22e (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251214 02:57:28.797065 20253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f95721f9c4489892d51ed0aa794585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.810989 20249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd1f7626cf406db2c278a08948befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.772022 20248 tablet_service.cc:1505] Processing CreateTablet for tablet bea937187ce744e99c7e48d4afc3469d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251214 02:57:28.817631 20250 tablet_service.cc:1505] Processing CreateTablet for tablet fcdae6c6dc2445d591d0bbefed18e4b9 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251214 02:57:28.818257 20247 tablet_service.cc:1505] Processing CreateTablet for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251214 02:57:28.822101 20252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9ad90d00ed4e249f54ebbe0b02e22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.826702 20250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcdae6c6dc2445d591d0bbefed18e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.843423 20247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.851011 20248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea937187ce744e99c7e48d4afc3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.868008 20181 tablet_service.cc:1505] Processing CreateTablet for tablet 770832072b40455b9a76ba3e907fb517 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251214 02:57:28.875104 20178 tablet_service.cc:1505] Processing CreateTablet for tablet cb9ad90d00ed4e249f54ebbe0b02e22e (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251214 02:57:28.875928 20181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770832072b40455b9a76ba3e907fb517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.877595 20180 tablet_service.cc:1505] Processing CreateTablet for tablet c411c9b2da7e4e2b83d854f004f13bfa (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251214 02:57:28.882297 20178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9ad90d00ed4e249f54ebbe0b02e22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.869570 20179 tablet_service.cc:1505] Processing CreateTablet for tablet 26f95721f9c4489892d51ed0aa794585 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251214 02:57:28.890647 20176 tablet_service.cc:1505] Processing CreateTablet for tablet fcdae6c6dc2445d591d0bbefed18e4b9 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251214 02:57:28.890776 20175 tablet_service.cc:1505] Processing CreateTablet for tablet 99fd1f7626cf406db2c278a08948befb (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251214 02:57:28.893451 20180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c411c9b2da7e4e2b83d854f004f13bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.888582 20177 tablet_service.cc:1505] Processing CreateTablet for tablet c55d8a42b8694ebf9b76fbb571a9814d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251214 02:57:28.899605 20173 tablet_service.cc:1505] Processing CreateTablet for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7 (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251214 02:57:28.901584 20174 tablet_service.cc:1505] Processing CreateTablet for tablet bea937187ce744e99c7e48d4afc3469d (DEFAULT_TABLE table=test-workload [id=305c39c1d966474da2e7f81205a7116b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251214 02:57:28.911672 20176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcdae6c6dc2445d591d0bbefed18e4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.913568 20179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f95721f9c4489892d51ed0aa794585. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.917934 20175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd1f7626cf406db2c278a08948befb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.921804 20173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d39ca6708704cd48f2c17fd7fdd6ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.923983 20177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55d8a42b8694ebf9b76fbb571a9814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.961918 20174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea937187ce744e99c7e48d4afc3469d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:28.992751 20308 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
W20251214 02:57:29.014000 20061 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:57:29.046123 20309 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.051631 20308 tablet_bootstrap.cc:654] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.079142 20308 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.079312 20310 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.079713 20308 ts_tablet_manager.cc:1403] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.006s
I20251214 02:57:29.089571 20308 raft_consensus.cc:359] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.091140 20308 raft_consensus.cc:385] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.091444 20308 raft_consensus.cc:740] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.092391 20308 consensus_queue.cc:260] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.098176 20309 tablet_bootstrap.cc:654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.098770 20141 heartbeater.cc:499] Master 127.19.131.126:44625 was elected leader, sending a full tablet report...
I20251214 02:57:29.100411 20308 ts_tablet_manager.cc:1434] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.020s user 0.008s sys 0.003s
I20251214 02:57:29.101524 20308 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.102248 20310 tablet_bootstrap.cc:654] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.107496 20308 tablet_bootstrap.cc:654] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.119989 20308 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.120504 20308 ts_tablet_manager.cc:1403] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20251214 02:57:29.125255 20310 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.125706 20310 ts_tablet_manager.cc:1403] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.007s
I20251214 02:57:29.126766 20308 raft_consensus.cc:359] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.127480 20308 raft_consensus.cc:385] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.127755 20308 raft_consensus.cc:740] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.128595 20310 raft_consensus.cc:359] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.128489 20308 consensus_queue.cc:260] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.129273 20310 raft_consensus.cc:385] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.129562 20310 raft_consensus.cc:740] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.130198 20310 consensus_queue.cc:260] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.130894 20308 ts_tablet_manager.cc:1434] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20251214 02:57:29.131840 20308 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.125963 20309 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.132382 20309 ts_tablet_manager.cc:1403] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.018s
I20251214 02:57:29.135052 20309 raft_consensus.cc:359] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.135728 20309 raft_consensus.cc:385] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.136013 20309 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.136662 20309 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.138211 20308 tablet_bootstrap.cc:654] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.140219 20216 heartbeater.cc:499] Master 127.19.131.126:44625 was elected leader, sending a full tablet report...
I20251214 02:57:29.141186 20290 heartbeater.cc:499] Master 127.19.131.126:44625 was elected leader, sending a full tablet report...
I20251214 02:57:29.144505 20310 ts_tablet_manager.cc:1434] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20251214 02:57:29.145534 20310 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.154673 20309 ts_tablet_manager.cc:1434] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20251214 02:57:29.155699 20309 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
W20251214 02:57:29.158627 20217 tablet.cc:2378] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 02:57:29.161388 20308 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.161844 20308 ts_tablet_manager.cc:1403] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.016s
I20251214 02:57:29.162812 20310 tablet_bootstrap.cc:654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.163275 20309 tablet_bootstrap.cc:654] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.165511 20308 raft_consensus.cc:359] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.166257 20308 raft_consensus.cc:385] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.166553 20308 raft_consensus.cc:740] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.167275 20308 consensus_queue.cc:260] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.169915 20308 ts_tablet_manager.cc:1434] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251214 02:57:29.170926 20308 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.177419 20308 tablet_bootstrap.cc:654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.183199 20310 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.183528 20310 ts_tablet_manager.cc:1403] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.023s
I20251214 02:57:29.185408 20310 raft_consensus.cc:359] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.185890 20310 raft_consensus.cc:385] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.186074 20310 raft_consensus.cc:740] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.186578 20310 consensus_queue.cc:260] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.188544 20308 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.188975 20308 ts_tablet_manager.cc:1403] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20251214 02:57:29.189158 20310 ts_tablet_manager.cc:1434] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251214 02:57:29.189998 20310 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.191630 20308 raft_consensus.cc:359] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.192348 20308 raft_consensus.cc:385] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.192636 20308 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.193382 20308 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.195631 20312 raft_consensus.cc:493] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.196053 20310 tablet_bootstrap.cc:654] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.196086 20312 raft_consensus.cc:515] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.199100 20312 leader_election.cc:290] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.200208 20308 ts_tablet_manager.cc:1434] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20251214 02:57:29.201170 20308 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.207619 20308 tablet_bootstrap.cc:654] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.228456 20309 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.228924 20309 ts_tablet_manager.cc:1403] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.024s
I20251214 02:57:29.231462 20309 raft_consensus.cc:359] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.232146 20309 raft_consensus.cc:385] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.232437 20309 raft_consensus.cc:740] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.233645 20309 consensus_queue.cc:260] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.238608 20309 ts_tablet_manager.cc:1434] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20251214 02:57:29.241153 20309 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.242488 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.244369 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
W20251214 02:57:29.246048 20082 leader_election.cc:343] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:57:29.246999 20082 leader_election.cc:343] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.247169 20310 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.247366 20082 leader_election.cc:304] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.247609 20310 ts_tablet_manager.cc:1403] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.058s user 0.016s sys 0.016s
I20251214 02:57:29.248183 20312 raft_consensus.cc:2749] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.250348 20310 raft_consensus.cc:359] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.251085 20310 raft_consensus.cc:385] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.251374 20310 raft_consensus.cc:740] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.251770 20309 tablet_bootstrap.cc:654] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.252061 20310 consensus_queue.cc:260] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.252632 20308 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.253087 20308 ts_tablet_manager.cc:1403] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.019s
I20251214 02:57:29.254521 20310 ts_tablet_manager.cc:1434] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:57:29.255551 20310 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.256474 20308 raft_consensus.cc:359] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.257191 20308 raft_consensus.cc:385] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.257515 20308 raft_consensus.cc:740] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.258210 20308 consensus_queue.cc:260] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.262946 20310 tablet_bootstrap.cc:654] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.260782 20308 ts_tablet_manager.cc:1434] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:57:29.265789 20308 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.267488 20312 raft_consensus.cc:493] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.267930 20312 raft_consensus.cc:515] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.268656 20309 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.269106 20309 ts_tablet_manager.cc:1403] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.011s
I20251214 02:57:29.269650 20310 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.269752 20312 leader_election.cc:290] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.270186 20310 ts_tablet_manager.cc:1403] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20251214 02:57:29.271262 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.271801 20309 raft_consensus.cc:359] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.272336 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.272529 20309 raft_consensus.cc:385] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.272876 20309 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.272755 20310 raft_consensus.cc:359] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.273554 20310 raft_consensus.cc:385] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.273926 20310 raft_consensus.cc:740] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.273604 20309 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.274711 20310 consensus_queue.cc:260] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.276628 20309 ts_tablet_manager.cc:1434] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20251214 02:57:29.277146 20310 ts_tablet_manager.cc:1434] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20251214 02:57:29.277568 20309 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.277999 20310 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
W20251214 02:57:29.279345 20082 leader_election.cc:343] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.273038 20265 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 0.
I20251214 02:57:29.280664 20082 leader_election.cc:304] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters: 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.281633 20312 raft_consensus.cc:2804] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.281962 20312 raft_consensus.cc:493] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.282262 20312 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.284827 20310 tablet_bootstrap.cc:654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.284838 20309 tablet_bootstrap.cc:654] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.295401 20312 raft_consensus.cc:515] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.297950 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
W20251214 02:57:29.299278 20082 leader_election.cc:343] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.300632 20308 tablet_bootstrap.cc:654] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.304141 20312 leader_election.cc:290] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.304665 20312 raft_consensus.cc:493] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.305239 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.305747 20265 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.305063 20312 raft_consensus.cc:515] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.309566 20312 leader_election.cc:290] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.310876 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.311141 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.311604 20264 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 0.
W20251214 02:57:29.312361 20082 leader_election.cc:343] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.312882 20082 leader_election.cc:304] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters: 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.313539 20312 raft_consensus.cc:2804] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.313892 20312 raft_consensus.cc:493] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.314210 20312 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.314620 20310 tablet_bootstrap.cc:492] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.315078 20310 ts_tablet_manager.cc:1403] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.013s
I20251214 02:57:29.315188 20309 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.315652 20309 ts_tablet_manager.cc:1403] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.038s user 0.031s sys 0.001s
I20251214 02:57:29.315143 20265 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 1.
I20251214 02:57:29.317279 20082 leader_election.cc:304] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters: 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.318998 20309 raft_consensus.cc:359] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.319705 20309 raft_consensus.cc:385] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.319437 20310 raft_consensus.cc:359] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.320027 20309 raft_consensus.cc:740] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.320176 20310 raft_consensus.cc:385] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.320482 20310 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.320761 20309 consensus_queue.cc:260] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.321349 20310 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.322702 20312 raft_consensus.cc:515] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.323570 20310 ts_tablet_manager.cc:1434] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251214 02:57:29.323570 20309 ts_tablet_manager.cc:1434] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251214 02:57:29.324579 20310 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.325991 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.327366 20309 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.331171 20310 tablet_bootstrap.cc:654] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
W20251214 02:57:29.333946 20082 leader_election.cc:343] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.334288 20309 tablet_bootstrap.cc:654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.325762 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.335240 20265 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.319509 20308 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.337771 20308 ts_tablet_manager.cc:1403] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.005s
I20251214 02:57:29.340480 20308 raft_consensus.cc:359] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.341161 20308 raft_consensus.cc:385] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.341466 20308 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.342145 20308 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.343451 20310 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.343956 20310 ts_tablet_manager.cc:1403] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20251214 02:57:29.344127 20312 leader_election.cc:290] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.344707 20312 raft_consensus.cc:2804] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.345125 20312 raft_consensus.cc:697] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:29.346029 20312 consensus_queue.cc:237] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.347069 20310 raft_consensus.cc:359] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.348037 20310 raft_consensus.cc:385] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.348379 20310 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.349176 20310 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.351989 20310 ts_tablet_manager.cc:1434] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251214 02:57:29.353102 20310 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.359742 20310 tablet_bootstrap.cc:654] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.366400 20265 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 1.
I20251214 02:57:29.372017 20308 ts_tablet_manager.cc:1434] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.034s user 0.009s sys 0.008s
I20251214 02:57:29.379567 20308 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.379549 20082 leader_election.cc:304] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters: 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.381067 20334 raft_consensus.cc:2804] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.381568 20334 raft_consensus.cc:697] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:29.374657 20310 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.382926 20310 ts_tablet_manager.cc:1403] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.003s
I20251214 02:57:29.382404 20334 consensus_queue.cc:237] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.384635 20309 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.385079 20309 ts_tablet_manager.cc:1403] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.058s user 0.032s sys 0.020s
I20251214 02:57:29.386150 20310 raft_consensus.cc:359] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.386922 20310 raft_consensus.cc:385] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.387213 20310 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.388022 20310 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.390414 20310 ts_tablet_manager.cc:1434] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:57:29.390864 20308 tablet_bootstrap.cc:654] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.391356 20310 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.394944 20316 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.395448 20316 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.388154 20309 raft_consensus.cc:359] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.397503 20309 raft_consensus.cc:385] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.397853 20309 raft_consensus.cc:740] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.398206 20316 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.398500 20309 consensus_queue.cc:260] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.401207 20309 ts_tablet_manager.cc:1434] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20251214 02:57:29.402294 20309 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.401278 20010 catalog_manager.cc:5654] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 1 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:29.409221 20309 tablet_bootstrap.cc:654] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.411522 20310 tablet_bootstrap.cc:654] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.416478 20317 raft_consensus.cc:493] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.417132 20317 raft_consensus.cc:515] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.422050 20316 raft_consensus.cc:493] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.426362 20317 leader_election.cc:290] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.427150 20316 raft_consensus.cc:515] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.434141 20316 leader_election.cc:290] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.439499 20316 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.463457 20316 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.467806 20317 raft_consensus.cc:493] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.473268 20316 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917)
I20251214 02:57:29.469458 20317 raft_consensus.cc:515] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.483362 20317 leader_election.cc:290] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.485107 20317 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.492550 20317 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.500301 20310 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.500756 20310 ts_tablet_manager.cc:1403] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.000s
I20251214 02:57:29.504684 20310 raft_consensus.cc:359] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.505285 20310 raft_consensus.cc:385] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.505481 20310 raft_consensus.cc:740] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.505929 20310 consensus_queue.cc:260] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.510636 20310 ts_tablet_manager.cc:1434] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251214 02:57:29.510835 20317 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.512604 20310 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Bootstrap starting.
I20251214 02:57:29.515628 20308 tablet_bootstrap.cc:492] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.516963 20308 ts_tablet_manager.cc:1403] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.138s user 0.030s sys 0.024s
I20251214 02:57:29.523757 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.524945 20116 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 0.
I20251214 02:57:29.526013 20308 raft_consensus.cc:359] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.526737 20308 raft_consensus.cc:385] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.527035 20308 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.527675 20308 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.530704 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.530541 20115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.532116 20116 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 0.
I20251214 02:57:29.534858 20229 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:29.543812 20316 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.544257 20316 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.544335 20344 raft_consensus.cc:493] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.544677 20316 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.553323 20316 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
W20251214 02:57:29.543993 20229 leader_election.cc:343] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.545105 20344 raft_consensus.cc:515] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.556748 20309 tablet_bootstrap.cc:492] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.556450 20229 leader_election.cc:304] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:29.558195 20316 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.559170 20316 raft_consensus.cc:2804] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.559509 20316 raft_consensus.cc:493] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.559855 20316 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.560073 20309 ts_tablet_manager.cc:1403] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.030s
I20251214 02:57:29.560376 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.560930 20116 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.562243 20334 raft_consensus.cc:493] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.563257 20334 raft_consensus.cc:515] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.563601 20309 raft_consensus.cc:359] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.564278 20309 raft_consensus.cc:385] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.565502 20334 leader_election.cc:290] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.564563 20309 raft_consensus.cc:740] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.567238 20309 consensus_queue.cc:260] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.570380 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.568398 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.574936 20191 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 0.
I20251214 02:57:29.576134 20082 leader_election.cc:304] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:29.576263 20010 catalog_manager.cc:5654] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 1 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:29.577328 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.577924 20264 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 0.
W20251214 02:57:29.578361 20082 leader_election.cc:343] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.578456 20334 raft_consensus.cc:2804] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.579255 20334 raft_consensus.cc:493] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.579838 20334 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.578884 20308 ts_tablet_manager.cc:1434] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.059s user 0.006s sys 0.003s
I20251214 02:57:29.581256 20308 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.586907 20344 leader_election.cc:290] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.587445 20115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.588114 20309 ts_tablet_manager.cc:1434] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20251214 02:57:29.588874 20115 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
W20251214 02:57:29.589022 20155 leader_election.cc:343] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:57:29.590165 20229 leader_election.cc:343] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.585341 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.590884 20114 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 0.
I20251214 02:57:29.586985 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.592782 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.593358 20264 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 0.
I20251214 02:57:29.593283 20155 leader_election.cc:304] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:29.592813 20309 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.594221 20310 tablet_bootstrap.cc:654] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.594362 20317 raft_consensus.cc:2804] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.596426 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.597311 20317 raft_consensus.cc:493] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.597436 20157 leader_election.cc:304] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters: 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.601922 20264 raft_consensus.cc:2393] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974a58ccac3b4728a98b46838595d05e in current term 1: Already voted for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in this term.
I20251214 02:57:29.598489 20317 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.612308 20116 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 1.
I20251214 02:57:29.612488 20334 raft_consensus.cc:515] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.613446 20229 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:29.614240 20344 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.619199 20317 raft_consensus.cc:515] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.621027 20317 leader_election.cc:290] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.621492 20317 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.624518 20351 raft_consensus.cc:2804] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.622777 20317 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.624364 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.626809 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.626940 20116 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.627735 20317 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.625847 20351 raft_consensus.cc:493] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.628633 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
W20251214 02:57:29.630391 20082 leader_election.cc:343] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:57:29.632006 20155 leader_election.cc:343] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.632694 20351 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.626202 20334 leader_election.cc:290] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.637434 20116 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 1.
I20251214 02:57:29.638574 20155 leader_election.cc:304] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:29.639662 20317 raft_consensus.cc:2804] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.640106 20317 raft_consensus.cc:697] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:29.640854 20317 consensus_queue.cc:237] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.646106 20344 raft_consensus.cc:697] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:29.648954 20353 raft_consensus.cc:493] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.642421 20351 raft_consensus.cc:515] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.651002 20308 tablet_bootstrap.cc:654] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.649401 20316 raft_consensus.cc:515] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
W20251214 02:57:29.654232 20157 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:35957, user_credentials={real_user=slave}} blocked reactor thread for 54351.5us
I20251214 02:57:29.655372 20310 tablet_bootstrap.cc:492] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: No bootstrap required, opened a new log
I20251214 02:57:29.655778 20310 ts_tablet_manager.cc:1403] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent bootstrapping tablet: real 0.144s user 0.020s sys 0.035s
I20251214 02:57:29.656824 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.657414 20264 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.662743 20316 leader_election.cc:290] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.663269 20309 tablet_bootstrap.cc:654] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.665268 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.665863 20265 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 0.
I20251214 02:57:29.665987 20310 raft_consensus.cc:359] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.653519 20344 consensus_queue.cc:237] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.666721 20310 raft_consensus.cc:385] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.667062 20310 raft_consensus.cc:740] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.668289 20310 consensus_queue.cc:260] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.649400 20353 raft_consensus.cc:515] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.672099 20310 ts_tablet_manager.cc:1434] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20251214 02:57:29.672300 20353 leader_election.cc:290] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.676882 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.678200 20116 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.678318 20351 leader_election.cc:290] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.679502 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.680030 20191 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.688516 20352 raft_consensus.cc:493] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.694442 20190 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.696911 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.697597 20190 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 0.
I20251214 02:57:29.698483 20189 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
W20251214 02:57:29.699437 20155 leader_election.cc:343] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.700974 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.691682 20188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.693115 20186 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.690467 20352 raft_consensus.cc:515] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.702600 20186 raft_consensus.cc:2393] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in current term 1: Already voted for candidate 974a58ccac3b4728a98b46838595d05e in this term.
I20251214 02:57:29.698606 20185 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.699688 20264 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 1.
I20251214 02:57:29.701972 20114 raft_consensus.cc:2468] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 0.
I20251214 02:57:29.703655 20187 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.706894 20188 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 0.
I20251214 02:57:29.708477 20187 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.701159 20157 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974a58ccac3b4728a98b46838595d05e; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.702795 20353 raft_consensus.cc:493] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:29.703759 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.711467 20116 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 1.
I20251214 02:57:29.711385 20157 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters: 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.711889 20353 raft_consensus.cc:515] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.709668 20352 leader_election.cc:290] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.712870 20351 raft_consensus.cc:2804] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.713449 20351 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.713781 20351 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.715987 20309 tablet_bootstrap.cc:492] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.716432 20309 ts_tablet_manager.cc:1403] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.002s
I20251214 02:57:29.716956 20187 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 1.
I20251214 02:57:29.717104 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:29.718899 20317 raft_consensus.cc:2749] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.717712 20229 leader_election.cc:304] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:29.733342 20351 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.734931 20353 leader_election.cc:290] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.735688 20229 leader_election.cc:304] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:29.735199 20351 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:29.804091 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
W20251214 02:57:29.804899 20229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:33917, user_credentials={real_user=slave}} blocked reactor thread for 69347.4us
I20251214 02:57:29.805258 20116 raft_consensus.cc:2393] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in current term 1: Already voted for candidate 9269ad98b10f4d61a2d9a943a117684d in this term.
I20251214 02:57:29.805594 20353 raft_consensus.cc:2804] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 02:57:29.803318 20157 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:35957, user_credentials={real_user=slave}} blocked reactor thread for 86093.6us
I20251214 02:57:29.805167 20309 raft_consensus.cc:359] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.803608 20308 tablet_bootstrap.cc:492] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.806209 20309 raft_consensus.cc:385] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.806294 20353 raft_consensus.cc:697] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:29.806591 20309 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.806463 20308 ts_tablet_manager.cc:1403] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.225s user 0.024s sys 0.030s
I20251214 02:57:29.807183 20353 consensus_queue.cc:237] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.808341 20264 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.735952 20010 catalog_manager.cc:5654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 1 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:29.808887 20264 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.810364 20308 raft_consensus.cc:359] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.807677 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.740556 20344 raft_consensus.cc:2804] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.737483 20191 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 1.
I20251214 02:57:29.811038 20308 raft_consensus.cc:385] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.811300 20344 raft_consensus.cc:493] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.811378 20308 raft_consensus.cc:740] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.811744 20344 raft_consensus.cc:3060] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.812059 20308 consensus_queue.cc:260] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.807274 20309 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.812849 20114 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:57:29.814791 20231 leader_election.cc:343] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.815665 20231 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters: 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.816686 20187 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e" is_pre_election: true
I20251214 02:57:29.817080 20355 raft_consensus.cc:2804] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:29.817478 20187 raft_consensus.cc:2393] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in current term 1: Already voted for candidate 9269ad98b10f4d61a2d9a943a117684d in this term.
I20251214 02:57:29.817415 20355 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:29.817662 20309 ts_tablet_manager.cc:1434] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.101s user 0.016s sys 0.010s
I20251214 02:57:29.817907 20355 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.818544 20082 leader_election.cc:304] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9
I20251214 02:57:29.819592 20334 raft_consensus.cc:2804] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.819623 20114 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 1.
I20251214 02:57:29.820223 20334 raft_consensus.cc:697] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:29.821082 20334 consensus_queue.cc:237] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.822949 20155 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:29.816747 20231 leader_election.cc:304] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9; no voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.824676 20344 raft_consensus.cc:515] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
W20251214 02:57:29.825204 20231 leader_election.cc:343] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:57:29.826925 20231 leader_election.cc:343] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.827065 20309 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.829190 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
W20251214 02:57:29.832794 20082 consensus_peers.cc:597] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d -> Peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Couldn't send request to peer 974a58ccac3b4728a98b46838595d05e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:57:29.823693 20351 raft_consensus.cc:2804] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.834532 20351 raft_consensus.cc:697] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:29.834702 20312 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:29.836185 20114 raft_consensus.cc:3060] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.840787 20308 ts_tablet_manager.cc:1434] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.032s user 0.004s sys 0.000s
I20251214 02:57:29.840503 20351 consensus_queue.cc:237] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.841717 20308 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: Bootstrap starting.
I20251214 02:57:29.836299 20231 leader_election.cc:304] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9; no voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.843353 20353 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.840901 20012 catalog_manager.cc:5654] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 0 to 1, leader changed from <none> to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 1 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:29.845693 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:29.845932 20264 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 1.
I20251214 02:57:29.845999 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.846731 20316 raft_consensus.cc:2749] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.847294 20157 leader_election.cc:304] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters: 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.848210 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:29.848927 20317 raft_consensus.cc:2804] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader election won for term 1
W20251214 02:57:29.849190 20231 leader_election.cc:343] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:29.849380 20317 raft_consensus.cc:697] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:29.849938 20157 leader_election.cc:304] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974a58ccac3b4728a98b46838595d05e; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:29.842618 20114 raft_consensus.cc:2468] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 1.
I20251214 02:57:29.850183 20317 consensus_queue.cc:237] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.851465 20229 leader_election.cc:304] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters: 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.853291 20352 raft_consensus.cc:2749] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.853938 20308 tablet_bootstrap.cc:654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.854296 20316 raft_consensus.cc:2804] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:29.854293 20011 catalog_manager.cc:5654] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 1 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:29.863545 20309 tablet_bootstrap.cc:654] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:29.866139 20344 leader_election.cc:290] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:29.879109 20355 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.880470 20263 raft_consensus.cc:2393] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 974a58ccac3b4728a98b46838595d05e in current term 1: Already voted for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in this term.
I20251214 02:57:29.885095 20353 raft_consensus.cc:2749] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.890544 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:29.891337 20114 raft_consensus.cc:2393] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in current term 1: Already voted for candidate 974a58ccac3b4728a98b46838595d05e in this term.
I20251214 02:57:29.892634 20355 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917)
I20251214 02:57:29.893200 20011 catalog_manager.cc:5654] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 1 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:29.896070 20334 consensus_queue.cc:1048] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:29.903784 20308 tablet_bootstrap.cc:492] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: No bootstrap required, opened a new log
I20251214 02:57:29.904229 20308 ts_tablet_manager.cc:1403] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.004s
I20251214 02:57:29.906612 20308 raft_consensus.cc:359] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.907250 20308 raft_consensus.cc:385] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.907505 20308 raft_consensus.cc:740] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.908218 20309 tablet_bootstrap.cc:492] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.908145 20308 consensus_queue.cc:260] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.908658 20309 ts_tablet_manager.cc:1403] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.009s
I20251214 02:57:29.909359 20316 raft_consensus.cc:697] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:29.910508 20308 ts_tablet_manager.cc:1434] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251214 02:57:29.911454 20309 raft_consensus.cc:359] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.912078 20309 raft_consensus.cc:385] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.912338 20309 raft_consensus.cc:740] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.910246 20316 consensus_queue.cc:237] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.912935 20309 consensus_queue.cc:260] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.917994 20309 ts_tablet_manager.cc:1434] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251214 02:57:29.918795 20309 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: Bootstrap starting.
I20251214 02:57:29.923326 20309 tablet_bootstrap.cc:654] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: Neither blocks nor log segments found. Creating new log.
W20251214 02:57:29.930899 20157 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:35957, user_credentials={real_user=slave}} blocked reactor thread for 81080.7us
I20251214 02:57:29.932452 20191 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:29.931941 20011 catalog_manager.cc:5654] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 1 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:29.941860 20312 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:29.943418 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:29.944816 20309 tablet_bootstrap.cc:492] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: No bootstrap required, opened a new log
I20251214 02:57:29.944847 20231 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9; no voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:29.945339 20309 ts_tablet_manager.cc:1403] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.008s
I20251214 02:57:29.946058 20316 raft_consensus.cc:2749] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:57:29.947891 20309 raft_consensus.cc:359] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.948792 20309 raft_consensus.cc:385] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:29.949036 20309 raft_consensus.cc:740] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Initialized, Role: FOLLOWER
I20251214 02:57:29.951989 20309 consensus_queue.cc:260] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:29.952282 20012 catalog_manager.cc:5654] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 0 to 1, leader changed from <none> to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 1 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:29.954210 20309 ts_tablet_manager.cc:1434] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251214 02:57:29.954030 20012 catalog_manager.cc:5654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 0 to 1, leader changed from <none> to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 1 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.018636 20191 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "770832072b40455b9a76ba3e907fb517"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39152
I20251214 02:57:30.019227 20191 raft_consensus.cc:606] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.024492 20191 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39152
I20251214 02:57:30.024973 20191 raft_consensus.cc:606] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.043013 20265 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:48384
I20251214 02:57:30.043496 20265 raft_consensus.cc:606] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.048079 20191 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39152
I20251214 02:57:30.048547 20191 raft_consensus.cc:606] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.060246 20114 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:30.060814 20114 raft_consensus.cc:606] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:30.065863 20265 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:48384
I20251214 02:57:30.066321 20265 raft_consensus.cc:606] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:30.070556 20114 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:30.071056 20114 raft_consensus.cc:606] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:30.075234 20114 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:30.075693 20114 raft_consensus.cc:606] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:30.101778 20363 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.110183 20317 raft_consensus.cc:993] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.111424 20317 raft_consensus.cc:1081] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.113543 20116 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:30.116773 20116 raft_consensus.cc:493] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.116715 20363 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:30.117282 20116 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.124006 20116 raft_consensus.cc:515] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.125013 20114 raft_consensus.cc:1240] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Rejecting Update request from peer 974a58ccac3b4728a98b46838595d05e for earlier term 1. Current term is 2. Ops: []
I20251214 02:57:30.128206 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:30.128633 20363 consensus_queue.cc:1059] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.129606 20116 leader_election.cc:290] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.129456 20191 raft_consensus.cc:3055] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.130456 20191 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.131244 20191 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.132318 20191 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.133004 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.133620 20265 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.139076 20191 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
W20251214 02:57:30.139892 20352 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20251214 02:57:30.139802 20265 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.140293 20082 leader_election.cc:304] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:30.141090 20312 raft_consensus.cc:2804] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.141561 20312 raft_consensus.cc:697] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.142184 20312 consensus_queue.cc:237] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.149158 20011 catalog_manager.cc:5654] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 1 to 2, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 2 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.166198 20316 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.167096 20352 raft_consensus.cc:493] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:30.167569 20352 raft_consensus.cc:515] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.169353 20352 leader_election.cc:290] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.170327 20116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:30.170760 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:30.172102 20155 leader_election.cc:304] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974a58ccac3b4728a98b46838595d05e; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.175308 20352 raft_consensus.cc:2749] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:30.176419 20365 raft_consensus.cc:993] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.176836 20365 raft_consensus.cc:1081] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.178757 20187 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39138
I20251214 02:57:30.179298 20187 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.179641 20187 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.182121 20352 raft_consensus.cc:493] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:30.182550 20352 raft_consensus.cc:515] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.184296 20352 leader_election.cc:290] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.186174 20316 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.186967 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" is_pre_election: true
I20251214 02:57:30.187563 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4d39ca6708704cd48f2c17fd7fdd6ae7" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269ad98b10f4d61a2d9a943a117684d" is_pre_election: true
I20251214 02:57:30.188277 20114 raft_consensus.cc:2393] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974a58ccac3b4728a98b46838595d05e in current term 1: Already voted for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in this term.
I20251214 02:57:30.189301 20155 leader_election.cc:304] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974a58ccac3b4728a98b46838595d05e; no voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.191953 20352 raft_consensus.cc:3060] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:30.191898 20316 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:30.196511 20187 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.198520 20191 raft_consensus.cc:1240] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Rejecting Update request from peer 4c28f10b4cbc48ab8f06564fc8a5edf9 for earlier term 1. Current term is 2. Ops: []
I20251214 02:57:30.199575 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:30.200440 20316 consensus_queue.cc:1059] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.201175 20316 raft_consensus.cc:3055] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.201977 20316 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:30.202229 20352 raft_consensus.cc:2749] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:30.202919 20316 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.203789 20312 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.204285 20316 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.211388 20355 raft_consensus.cc:993] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.211829 20355 raft_consensus.cc:1081] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.213788 20114 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54836
I20251214 02:57:30.214320 20114 raft_consensus.cc:493] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.214691 20114 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.220247 20184 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:30.226853 20114 raft_consensus.cc:515] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.227988 20113 raft_consensus.cc:1240] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Rejecting Update request from peer 4c28f10b4cbc48ab8f06564fc8a5edf9 for earlier term 1. Current term is 2. Ops: []
I20251214 02:57:30.229110 20344 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.230003 20365 consensus_queue.cc:1059] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.230870 20365 raft_consensus.cc:3055] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.231197 20365 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:30.231943 20365 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.233023 20365 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.238715 20334 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.239996 20114 leader_election.cc:290] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.241073 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.244655 20187 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.246644 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.247866 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:30.258348 20263 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.258299 20265 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.260090 20157 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters: 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:30.260252 20082 leader_election.cc:304] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:30.260720 20352 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.261099 20334 raft_consensus.cc:2804] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.261665 20334 raft_consensus.cc:697] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.261144 20352 raft_consensus.cc:697] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.262470 20334 consensus_queue.cc:237] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.263747 20352 consensus_queue.cc:237] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.270087 20312 raft_consensus.cc:993] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.270509 20312 raft_consensus.cc:1081] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.272012 20191 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:30.272538 20191 raft_consensus.cc:493] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.272867 20191 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.275964 20011 catalog_manager.cc:5654] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 1 to 2, leader changed from 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 2 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.278985 20191 raft_consensus.cc:515] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.281622 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:30.282153 20114 raft_consensus.cc:3055] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.282461 20114 raft_consensus.cc:740] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.283185 20114 consensus_queue.cc:260] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.284183 20114 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.287438 20191 leader_election.cc:290] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.286736 20012 catalog_manager.cc:5654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 1 to 2, leader changed from 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 2 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.289523 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.290071 20263 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.290972 20114 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.292011 20155 leader_election.cc:304] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:30.292807 20352 raft_consensus.cc:2804] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.293242 20352 raft_consensus.cc:697] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.294005 20352 consensus_queue.cc:237] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.296911 20263 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.302933 20012 catalog_manager.cc:5654] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 1 to 2, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 2 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.311039 20114 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:30.312608 20363 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.320075 20363 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.338255 20363 raft_consensus.cc:993] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.338709 20363 raft_consensus.cc:1081] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.340087 20114 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:30.340600 20114 raft_consensus.cc:493] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.340907 20114 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.346380 20114 raft_consensus.cc:515] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.347340 20316 consensus_queue.cc:1048] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.348520 20114 leader_election.cc:290] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.349848 20113 raft_consensus.cc:1240] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Rejecting Update request from peer 974a58ccac3b4728a98b46838595d05e for earlier term 1. Current term is 2. Ops: []
I20251214 02:57:30.350369 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.350420 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:30.351001 20263 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.351387 20185 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:30.351071 20191 raft_consensus.cc:3055] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.352170 20191 raft_consensus.cc:740] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.352967 20191 consensus_queue.cc:260] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.357414 20352 consensus_queue.cc:1059] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:30.361670 20263 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.362911 20082 leader_election.cc:304] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:30.363755 20334 raft_consensus.cc:2804] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.364202 20334 raft_consensus.cc:697] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.365233 20334 consensus_queue.cc:237] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.364176 20191 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.376657 20191 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.376295 20012 catalog_manager.cc:5654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 1 to 2, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 2 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.390373 20351 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:30.397652 20316 consensus_queue.cc:1048] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:30.398170 20334 raft_consensus.cc:993] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.398814 20330 raft_consensus.cc:1081] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.400417 20191 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:30.400949 20191 raft_consensus.cc:493] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.401306 20191 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.407809 20191 raft_consensus.cc:515] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.410159 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:30.410315 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.410822 20114 raft_consensus.cc:3055] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.411253 20191 leader_election.cc:290] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.410842 20265 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.411191 20114 raft_consensus.cc:740] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.422425 20114 consensus_queue.cc:260] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.423904 20114 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.427654 20352 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:30.432183 20114 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.435876 20265 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.439687 20155 leader_election.cc:304] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:30.440630 20352 raft_consensus.cc:2804] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.441143 20352 raft_consensus.cc:697] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.441989 20352 consensus_queue.cc:237] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.452338 20010 catalog_manager.cc:5654] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 1 to 2, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 2 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.454164 20352 raft_consensus.cc:993] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.454643 20352 raft_consensus.cc:1081] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:30.456396 20113 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "770832072b40455b9a76ba3e907fb517"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:30.456947 20113 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.457319 20113 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.462903 20113 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.464792 20113 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.466233 20191 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:30.466866 20191 raft_consensus.cc:3055] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.467182 20191 raft_consensus.cc:740] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.467234 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.467751 20265 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.467980 20191 consensus_queue.cc:260] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.469224 20191 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.486732 20265 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.487769 20082 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:30.488530 20330 raft_consensus.cc:2804] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.488906 20330 raft_consensus.cc:697] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.489594 20330 consensus_queue.cc:237] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.490387 20191 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 2.
I20251214 02:57:30.496803 20013 catalog_manager.cc:5654] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 1 to 2, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 2 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.555178 20312 raft_consensus.cc:993] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.555641 20330 raft_consensus.cc:1081] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:30.557233 20191 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:30.557802 20191 raft_consensus.cc:493] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:30.558166 20191 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.563356 20191 raft_consensus.cc:515] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.564873 20191 leader_election.cc:290] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:30.565631 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:30.565935 20265 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:30.566215 20113 raft_consensus.cc:3055] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:30.566468 20265 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.566548 20113 raft_consensus.cc:740] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:30.567346 20113 consensus_queue.cc:260] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.568514 20113 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.573068 20265 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.574115 20157 leader_election.cc:304] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:30.574937 20113 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 2.
I20251214 02:57:30.574931 20352 raft_consensus.cc:2804] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:30.575556 20352 raft_consensus.cc:697] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:30.576378 20352 consensus_queue.cc:237] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:30.583079 20013 catalog_manager.cc:5654] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 1 to 2, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 2 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:30.603912 20191 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.605224 20330 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.615025 20265 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.617280 20330 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.665525 20265 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.666705 20352 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.674077 20113 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.680361 20113 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.681845 20351 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.739010 20265 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.740236 20330 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.747972 20191 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:30.758006 20191 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.759780 20334 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.815481 20265 raft_consensus.cc:1275] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.816673 20394 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.824152 20113 raft_consensus.cc:1275] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.825469 20394 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.852463 20113 raft_consensus.cc:1275] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.853838 20394 consensus_queue.cc:1048] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.866987 20265 raft_consensus.cc:1275] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.870162 20394 consensus_queue.cc:1048] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.876559 20263 raft_consensus.cc:1275] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.878423 20334 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:30.888862 20191 raft_consensus.cc:1275] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:30.892756 20330 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:31.006597 20263 raft_consensus.cc:1275] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:31.008013 20394 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:31.016451 20113 raft_consensus.cc:1275] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:31.018131 20394 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:31.057698 20265 raft_consensus.cc:1275] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:31.058941 20330 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:31.066112 20191 raft_consensus.cc:1275] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:31.067588 20312 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:57:32.699882 20044 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 19984
W20251214 02:57:32.700606 20044 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 20047
W20251214 02:57:32.701303 20044 debug-util.cc:398] Leaking SignalData structure 0x7b08001d41c0 after lost signal to thread 20140
W20251214 02:57:32.701974 20044 debug-util.cc:398] Leaking SignalData structure 0x7b080010a760 after lost signal to thread 20215
W20251214 02:57:32.702610 20044 debug-util.cc:398] Leaking SignalData structure 0x7b080010df00 after lost signal to thread 20289
I20251214 02:57:33.239586 19981 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20251214 02:57:33.240310 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:33.255412 20113 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:54864
I20251214 02:57:33.255892 20113 raft_consensus.cc:606] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Received request to transfer leadership to TS 4c28f10b4cbc48ab8f06564fc8a5edf9
I20251214 02:57:33.270723 20184 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:33.271342 20184 raft_consensus.cc:606] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Received request to transfer leadership to TS 4c28f10b4cbc48ab8f06564fc8a5edf9
I20251214 02:57:33.272500 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251214 02:57:33.272923 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:33.423787 20403 raft_consensus.cc:993] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:33.424257 20334 raft_consensus.cc:1081] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Signalling peer 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:33.425789 20265 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:48376
I20251214 02:57:33.426242 20265 raft_consensus.cc:493] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:33.426519 20265 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.431810 20265 raft_consensus.cc:515] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.434062 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:33.434298 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:33.433705 20265 leader_election.cc:290] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:33.434669 20113 raft_consensus.cc:3055] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:33.434901 20184 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.435084 20113 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:33.435838 20113 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.437011 20113 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.441411 20184 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:33.442715 20231 leader_election.cc:304] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:33.443396 20113 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:33.443529 20396 raft_consensus.cc:2804] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:33.444911 20396 raft_consensus.cc:697] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:33.445771 20396 consensus_queue.cc:237] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.448671 20405 raft_consensus.cc:993] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:33.449146 20405 raft_consensus.cc:1081] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Signalling peer 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:33.450917 20265 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:48380
I20251214 02:57:33.451371 20265 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:33.451627 20265 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.455956 20010 catalog_manager.cc:5654] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 2 to 3, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 3 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:33.456687 20265 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.458421 20265 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:33.459277 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:33.459412 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:33.459853 20113 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.459975 20184 raft_consensus.cc:3055] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:33.460389 20184 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:33.461169 20184 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.462461 20184 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:33.466255 20113 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:33.467677 20229 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:33.468585 20396 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:33.469043 20396 raft_consensus.cc:697] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:33.469851 20184 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:33.469808 20396 consensus_queue.cc:237] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:33.497259 20010 catalog_manager.cc:5654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 2 to 3, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 3 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:33.892532 20184 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:33.893669 20404 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:33.916337 20113 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:33.917785 20404 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:33.927178 20185 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:33.928584 20404 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:33.953114 20113 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:33.955556 20404 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:57:34.273737 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:34.278654 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:34.279179 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:35.280118 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:35.284912 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:35.285276 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:36.286106 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:36.291249 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:36.291648 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:37.292661 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:37.297729 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:37.298142 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:38.299118 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:38.304693 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:38.305068 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:39.305914 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:39.310901 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:39.311311 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:40.312100 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:40.317545 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:40.317939 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:41.318807 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:41.323472 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:41.323895 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:42.324788 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:42.329481 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:42.329875 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:43.355494 19981 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20251214 02:57:43.359653 20113 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "770832072b40455b9a76ba3e907fb517"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:43.360225 20113 raft_consensus.cc:606] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:43.364611 20263 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:48384
I20251214 02:57:43.365099 20263 raft_consensus.cc:606] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:43.369143 20113 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:43.369601 20113 raft_consensus.cc:606] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:43.373462 20113 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:54856
I20251214 02:57:43.373858 20113 raft_consensus.cc:606] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:43.380016 20263 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
mode: GRACEFUL
new_leader_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:48384
I20251214 02:57:43.380498 20263 raft_consensus.cc:606] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Received request to transfer leadership to TS 974a58ccac3b4728a98b46838595d05e
I20251214 02:57:43.382141 20427 raft_consensus.cc:993] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.382750 20429 raft_consensus.cc:1081] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.384205 20184 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:43.384647 20184 raft_consensus.cc:493] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:43.384964 20184 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.425799 20184 raft_consensus.cc:515] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.427898 20184 leader_election.cc:290] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:43.428854 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:43.429224 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:43.429607 20113 raft_consensus.cc:3055] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:43.429764 20263 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.429935 20113 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:43.430666 20113 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.431547 20113 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.504879 20415 raft_consensus.cc:993] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.505379 20430 raft_consensus.cc:1081] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.506680 20184 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39138
I20251214 02:57:43.507089 20184 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:43.507347 20184 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.527751 20113 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.527751 20263 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.527786 20184 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.529393 20155 leader_election.cc:304] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:43.529873 20184 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 4 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:43.530759 20425 raft_consensus.cc:2804] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:43.531494 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:43.531735 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:43.532106 20113 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.531913 20425 raft_consensus.cc:697] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:43.532495 20263 raft_consensus.cc:3055] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:57:43.532833 20263 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:43.533124 20425 consensus_queue.cc:237] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.533589 20263 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.534958 20263 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.542061 20013 catalog_manager.cc:5654] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 2 to 3, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 3 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:43.624029 20113 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 4.
I20251214 02:57:43.624045 20263 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 4.
I20251214 02:57:43.625312 20155 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:43.626103 20425 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:57:43.626459 20425 raft_consensus.cc:697] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:43.627283 20425 consensus_queue.cc:237] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.633807 20013 catalog_manager.cc:5654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 3 to 4, leader changed from 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 4 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:43.667472 20427 raft_consensus.cc:993] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.667892 20429 raft_consensus.cc:1081] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.669140 20184 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:43.669552 20184 raft_consensus.cc:493] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:43.669821 20184 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.723464 20184 raft_consensus.cc:515] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.724831 20184 leader_election.cc:290] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:43.725782 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:43.725874 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb9ad90d00ed4e249f54ebbe0b02e22e" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:43.726323 20113 raft_consensus.cc:3055] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:43.726440 20263 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.726626 20113 raft_consensus.cc:740] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:43.727249 20113 consensus_queue.cc:260] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.728060 20113 raft_consensus.cc:3060] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.743670 20427 raft_consensus.cc:993] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.744104 20429 raft_consensus.cc:1081] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.745352 20184 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "770832072b40455b9a76ba3e907fb517"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39126
I20251214 02:57:43.745785 20184 raft_consensus.cc:493] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:43.746079 20184 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.763118 20415 raft_consensus.cc:993] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: : Instructing follower 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.763515 20430 raft_consensus.cc:1081] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Signalling peer 974a58ccac3b4728a98b46838595d05e to start an election
I20251214 02:57:43.764851 20185 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
from {username='slave'} at 127.0.0.1:39138
I20251214 02:57:43.765290 20185 raft_consensus.cc:493] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:43.765558 20185 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.825529 20113 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.825534 20263 raft_consensus.cc:2468] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.827189 20157 leader_election.cc:304] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:43.827314 20184 raft_consensus.cc:515] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.828708 20425 raft_consensus.cc:2804] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:43.829946 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:43.830125 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "770832072b40455b9a76ba3e907fb517" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:43.829499 20184 leader_election.cc:290] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:43.830482 20113 raft_consensus.cc:3055] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:43.830711 20263 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.831005 20113 raft_consensus.cc:740] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:43.830662 20425 raft_consensus.cc:697] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:43.831960 20113 consensus_queue.cc:260] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.832204 20425 consensus_queue.cc:237] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.833135 20113 raft_consensus.cc:3060] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:43.839567 20013 catalog_manager.cc:5654] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 2 to 3, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 3 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:43.926023 20113 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.926061 20185 raft_consensus.cc:515] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.926414 20263 raft_consensus.cc:2468] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 3.
I20251214 02:57:43.927459 20155 leader_election.cc:304] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:43.928136 20185 leader_election.cc:290] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 4 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:43.929090 20425 raft_consensus.cc:2804] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:43.929761 20425 raft_consensus.cc:697] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:43.930713 20425 consensus_queue.cc:237] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.931058 20114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:43.931717 20114 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.931926 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c411c9b2da7e4e2b83d854f004f13bfa" candidate_uuid: "974a58ccac3b4728a98b46838595d05e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:43.932454 20263 raft_consensus.cc:3055] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:57:43.932747 20263 raft_consensus.cc:740] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:43.933298 20263 consensus_queue.cc:260] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:43.934127 20263 raft_consensus.cc:3060] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:43.942157 20013 catalog_manager.cc:5654] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 2 to 3, leader changed from 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 3 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:44.028115 20113 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:44.028988 20114 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 4.
I20251214 02:57:44.029304 20425 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:57:44.030257 20155 leader_election.cc:304] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:44.029235 20263 raft_consensus.cc:2468] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974a58ccac3b4728a98b46838595d05e in term 4.
I20251214 02:57:44.032061 20425 raft_consensus.cc:2804] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:57:44.032567 20425 raft_consensus.cc:697] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Becoming Leader. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:44.033416 20425 consensus_queue.cc:237] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:44.043601 20263 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:44.044327 20011 catalog_manager.cc:5654] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e reported cstate change: term changed from 3 to 4, leader changed from 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67) to 974a58ccac3b4728a98b46838595d05e (127.19.131.66). New cstate: current_term: 4 leader_uuid: "974a58ccac3b4728a98b46838595d05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:44.045156 20425 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:57:44.085748 20263 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.087343 20425 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:44.096138 20114 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.097440 20425 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:44.215233 20114 raft_consensus.cc:1275] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.216500 20425 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:44.225132 20263 raft_consensus.cc:1275] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.226645 20442 consensus_queue.cc:1048] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:57:44.389046 20263 raft_consensus.cc:1275] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.390094 20450 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:44.398404 20114 raft_consensus.cc:1275] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:44.399816 20450 consensus_queue.cc:1048] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:44.429461 20114 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:44.430583 20450 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:57:44.441824 20263 raft_consensus.cc:1275] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Refusing update from remote peer 974a58ccac3b4728a98b46838595d05e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:44.443626 20431 consensus_queue.cc:1048] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:57:46.414842 19981 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20251214 02:57:46.415477 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:46.421990 20185 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:46.422497 20185 raft_consensus.cc:606] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:46.424669 20185 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:46.425140 20185 raft_consensus.cc:606] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:46.427194 20185 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:46.427699 20185 raft_consensus.cc:606] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Received request to transfer leadership to TS 4c28f10b4cbc48ab8f06564fc8a5edf9
I20251214 02:57:46.428718 20457 raft_consensus.cc:993] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.429177 20456 raft_consensus.cc:1081] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.429970 20185 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:46.430521 20185 raft_consensus.cc:606] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Received request to transfer leadership to TS 4c28f10b4cbc48ab8f06564fc8a5edf9
I20251214 02:57:46.430680 20114 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:46.431210 20114 raft_consensus.cc:493] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:46.431530 20114 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:46.432763 20185 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "974a58ccac3b4728a98b46838595d05e"
mode: GRACEFUL
new_leader_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:39168
I20251214 02:57:46.433153 20185 raft_consensus.cc:606] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Received request to transfer leadership to TS 9269ad98b10f4d61a2d9a943a117684d
I20251214 02:57:46.433990 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:57:46.434370 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:46.435835 20114 raft_consensus.cc:515] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.437448 20114 leader_election.cc:290] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 4 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:46.438254 20185 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:46.438349 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26f95721f9c4489892d51ed0aa794585" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:46.438758 20185 raft_consensus.cc:3055] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:57:46.438859 20263 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:46.439030 20185 raft_consensus.cc:740] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:46.439777 20185 consensus_queue.cc:260] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.440698 20185 raft_consensus.cc:3060] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:57:46.443146 20263 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 4.
I20251214 02:57:46.444311 20082 leader_election.cc:304] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:46.445680 20458 raft_consensus.cc:2804] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:57:46.446146 20185 raft_consensus.cc:2468] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 4.
I20251214 02:57:46.446633 20458 raft_consensus.cc:697] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 4 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:46.447597 20458 consensus_queue.cc:237] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.453256 20011 catalog_manager.cc:5654] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 3 to 4, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 4 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:46.558063 20457 raft_consensus.cc:993] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:46.558650 20457 raft_consensus.cc:1081] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Signalling peer 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:46.559968 20465 raft_consensus.cc:993] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.559976 20263 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:48380
I20251214 02:57:46.560475 20465 raft_consensus.cc:1081] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.560525 20263 raft_consensus.cc:493] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:46.560860 20263 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.561866 20114 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:46.562263 20114 raft_consensus.cc:493] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:46.562510 20114 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.565915 20263 raft_consensus.cc:515] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.567471 20114 raft_consensus.cc:515] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.568248 20263 leader_election.cc:290] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:46.568809 20185 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:46.568802 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bea937187ce744e99c7e48d4afc3469d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:46.569463 20113 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.569384 20185 raft_consensus.cc:3055] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:46.569895 20185 raft_consensus.cc:740] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:46.570144 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:46.570669 20184 raft_consensus.cc:3055] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:46.570489 20185 consensus_queue.cc:260] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.570957 20184 raft_consensus.cc:740] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:46.571466 20185 raft_consensus.cc:3060] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.571703 20184 consensus_queue.cc:260] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.572682 20184 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.574659 20114 leader_election.cc:290] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 3 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:46.575989 20113 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:46.576331 20185 raft_consensus.cc:2468] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:46.576459 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99fd1f7626cf406db2c278a08948befb" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:46.576987 20263 raft_consensus.cc:3060] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.577407 20229 leader_election.cc:304] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:46.578533 20430 raft_consensus.cc:2804] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:46.579378 20184 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 3.
I20251214 02:57:46.580493 20082 leader_election.cc:304] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9269ad98b10f4d61a2d9a943a117684d, 974a58ccac3b4728a98b46838595d05e; no voters:
I20251214 02:57:46.581228 20263 raft_consensus.cc:2468] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 3.
I20251214 02:57:46.581282 20458 raft_consensus.cc:2804] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:46.581970 20458 raft_consensus.cc:697] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 3 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:46.581679 20430 raft_consensus.cc:697] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:46.582808 20458 consensus_queue.cc:237] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.583243 20430 consensus_queue.cc:237] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.590106 20013 catalog_manager.cc:5654] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 2 to 3, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 3 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:46.590715 20011 catalog_manager.cc:5654] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 2 to 3, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 3 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:46.799419 20457 raft_consensus.cc:993] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:46.800052 20465 raft_consensus.cc:1081] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Signalling peer 4c28f10b4cbc48ab8f06564fc8a5edf9 to start an election
I20251214 02:57:46.801690 20263 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d"
dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
from {username='slave'} at 127.0.0.1:48380
I20251214 02:57:46.802248 20263 raft_consensus.cc:493] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:46.802539 20263 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.807777 20263 raft_consensus.cc:515] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.809312 20263 leader_election.cc:290] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Requested vote from peers 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917), 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649)
I20251214 02:57:46.810950 20113 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
I20251214 02:57:46.810987 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c55d8a42b8694ebf9b76fbb571a9814d" candidate_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:46.811643 20113 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.811664 20184 raft_consensus.cc:3055] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:57:46.812036 20184 raft_consensus.cc:740] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:46.812718 20184 consensus_queue.cc:260] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.813812 20184 raft_consensus.cc:3060] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:57:46.816514 20113 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:46.817621 20229 leader_election.cc:304] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:46.818696 20430 raft_consensus.cc:2804] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:57:46.818851 20184 raft_consensus.cc:2468] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c28f10b4cbc48ab8f06564fc8a5edf9 in term 3.
I20251214 02:57:46.819077 20430 raft_consensus.cc:697] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Becoming Leader. State: Replica: 4c28f10b4cbc48ab8f06564fc8a5edf9, State: Running, Role: LEADER
I20251214 02:57:46.820091 20430 consensus_queue.cc:237] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.828765 20011 catalog_manager.cc:5654] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 reported cstate change: term changed from 2 to 3, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67). New cstate: current_term: 3 leader_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:46.912675 20457 raft_consensus.cc:993] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: : Instructing follower 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.913077 20465 raft_consensus.cc:1081] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Signalling peer 9269ad98b10f4d61a2d9a943a117684d to start an election
I20251214 02:57:46.914430 20113 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9"
dest_uuid: "9269ad98b10f4d61a2d9a943a117684d"
from {username='slave'} at 127.0.0.1:54842
I20251214 02:57:46.914834 20113 raft_consensus.cc:493] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:46.915043 20113 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:57:46.919021 20113 raft_consensus.cc:515] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.920439 20113 leader_election.cc:290] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 5 election: Requested vote from peers 974a58ccac3b4728a98b46838595d05e (127.19.131.66:38649), 4c28f10b4cbc48ab8f06564fc8a5edf9 (127.19.131.67:35957)
I20251214 02:57:46.921365 20184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "974a58ccac3b4728a98b46838595d05e"
I20251214 02:57:46.921550 20263 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcdae6c6dc2445d591d0bbefed18e4b9" candidate_uuid: "9269ad98b10f4d61a2d9a943a117684d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9"
I20251214 02:57:46.921883 20184 raft_consensus.cc:3055] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:57:46.922024 20263 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:57:46.922202 20184 raft_consensus.cc:740] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974a58ccac3b4728a98b46838595d05e, State: Running, Role: LEADER
I20251214 02:57:46.922998 20184 consensus_queue.cc:260] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.924196 20184 raft_consensus.cc:3060] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:57:46.927960 20263 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 5.
I20251214 02:57:46.929090 20082 leader_election.cc:304] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c28f10b4cbc48ab8f06564fc8a5edf9, 9269ad98b10f4d61a2d9a943a117684d; no voters:
I20251214 02:57:46.929606 20184 raft_consensus.cc:2468] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269ad98b10f4d61a2d9a943a117684d in term 5.
I20251214 02:57:46.929898 20458 raft_consensus.cc:2804] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:57:46.930315 20458 raft_consensus.cc:697] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 5 LEADER]: Becoming Leader. State: Replica: 9269ad98b10f4d61a2d9a943a117684d, State: Running, Role: LEADER
I20251214 02:57:46.931149 20458 consensus_queue.cc:237] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } }
I20251214 02:57:46.938474 20011 catalog_manager.cc:5654] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d reported cstate change: term changed from 4 to 5, leader changed from 974a58ccac3b4728a98b46838595d05e (127.19.131.66) to 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65). New cstate: current_term: 5 leader_uuid: "9269ad98b10f4d61a2d9a943a117684d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:46.939301 20184 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:46.940531 20458 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:57:46.951016 20263 raft_consensus.cc:1275] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:57:46.952734 20475 consensus_queue.cc:1048] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:57:46.995826 20263 raft_consensus.cc:1275] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:46.996894 20475 consensus_queue.cc:1048] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:47.005704 20184 raft_consensus.cc:1275] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:47.007468 20459 consensus_queue.cc:1048] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:57:47.164160 20113 raft_consensus.cc:1275] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:47.164678 20184 raft_consensus.cc:1275] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:47.165225 20430 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:47.165920 20466 consensus_queue.cc:1048] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:47.215970 20113 raft_consensus.cc:1275] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:47.217047 20482 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269ad98b10f4d61a2d9a943a117684d" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:47.224962 20184 raft_consensus.cc:1275] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Refusing update from remote peer 4c28f10b4cbc48ab8f06564fc8a5edf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:57:47.226487 20466 consensus_queue.cc:1048] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:57:47.313674 20184 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 5 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:57:47.314702 20459 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "974a58ccac3b4728a98b46838595d05e" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:57:47.322750 20263 raft_consensus.cc:1275] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 5 FOLLOWER]: Refusing update from remote peer 9269ad98b10f4d61a2d9a943a117684d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:57:47.324220 20486 consensus_queue.cc:1048] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c28f10b4cbc48ab8f06564fc8a5edf9" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:57:47.435738 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:47.440668 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:47.441092 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:48.442175 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:48.446960 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:48.447391 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:49.448426 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:49.453729 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:49.454180 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:50.455324 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:50.460624 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:50.461087 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:51.462105 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:51.467053 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:51.467509 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:52.468425 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:52.473176 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:52.473599 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:53.474526 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:53.479279 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:53.479727 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:54.480741 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:54.485659 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:54.486079 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:55.487098 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:57:55.491922 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:57:55.492352 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:57:56.518076 19981 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20251214 02:57:56.518673 19981 tablet_server.cc:178] TabletServer@127.19.131.65:0 shutting down...
I20251214 02:57:56.549842 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:57:56.550617 19981 tablet_replica.cc:333] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.551370 19981 raft_consensus.cc:2243] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.551899 19981 raft_consensus.cc:2272] T 770832072b40455b9a76ba3e907fb517 P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.554380 19981 tablet_replica.cc:333] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.554909 19981 raft_consensus.cc:2243] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.555626 19981 raft_consensus.cc:2272] T 99fd1f7626cf406db2c278a08948befb P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.557199 19981 tablet_replica.cc:333] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.557684 19981 raft_consensus.cc:2243] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 5 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.558388 19981 raft_consensus.cc:2272] T fcdae6c6dc2445d591d0bbefed18e4b9 P 9269ad98b10f4d61a2d9a943a117684d [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.560036 19981 tablet_replica.cc:333] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.560520 19981 raft_consensus.cc:2243] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.560926 19981 raft_consensus.cc:2272] T c411c9b2da7e4e2b83d854f004f13bfa P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.562484 19981 tablet_replica.cc:333] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.563033 19981 raft_consensus.cc:2243] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.563390 19981 raft_consensus.cc:2272] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 9269ad98b10f4d61a2d9a943a117684d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.565096 19981 tablet_replica.cc:333] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.565757 19981 raft_consensus.cc:2243] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 4 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.566608 19981 raft_consensus.cc:2272] T 26f95721f9c4489892d51ed0aa794585 P 9269ad98b10f4d61a2d9a943a117684d [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.568483 19981 tablet_replica.cc:333] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.568945 19981 raft_consensus.cc:2243] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.569314 19981 raft_consensus.cc:2272] T bea937187ce744e99c7e48d4afc3469d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.570772 19981 tablet_replica.cc:333] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.571203 19981 raft_consensus.cc:2243] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.571574 19981 raft_consensus.cc:2272] T cb9ad90d00ed4e249f54ebbe0b02e22e P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.573098 19981 tablet_replica.cc:333] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d: stopping tablet replica
I20251214 02:57:56.573496 19981 raft_consensus.cc:2243] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.573906 19981 raft_consensus.cc:2272] T c55d8a42b8694ebf9b76fbb571a9814d P 9269ad98b10f4d61a2d9a943a117684d [term 3 FOLLOWER]: Raft consensus is shut down!
W20251214 02:57:56.651221 20229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.131.65:33917: connect: Connection refused (error 111)
I20251214 02:57:56.655261 19981 tablet_server.cc:195] TabletServer@127.19.131.65:0 shutdown complete.
W20251214 02:57:56.655336 20229 consensus_peers.cc:597] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 -> Peer 9269ad98b10f4d61a2d9a943a117684d (127.19.131.65:33917): Couldn't send request to peer 9269ad98b10f4d61a2d9a943a117684d. Status: Network error: Client connection negotiation failed: client connection to 127.19.131.65:33917: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251214 02:57:56.672969 19981 tablet_server.cc:178] TabletServer@127.19.131.66:0 shutting down...
I20251214 02:57:56.698083 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:57:56.698750 19981 tablet_replica.cc:333] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.699314 19981 raft_consensus.cc:2243] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.699782 19981 raft_consensus.cc:2272] T 99fd1f7626cf406db2c278a08948befb P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.701445 19981 tablet_replica.cc:333] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.701972 19981 raft_consensus.cc:2243] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.702364 19981 raft_consensus.cc:2272] T 26f95721f9c4489892d51ed0aa794585 P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.703940 19981 tablet_replica.cc:333] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.704393 19981 raft_consensus.cc:2243] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.704788 19981 raft_consensus.cc:2272] T bea937187ce744e99c7e48d4afc3469d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.706262 19981 tablet_replica.cc:333] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.706782 19981 raft_consensus.cc:2243] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.707155 19981 raft_consensus.cc:2272] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 974a58ccac3b4728a98b46838595d05e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.708981 19981 tablet_replica.cc:333] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.709476 19981 raft_consensus.cc:2243] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 4 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.710299 19981 raft_consensus.cc:2272] T c411c9b2da7e4e2b83d854f004f13bfa P 974a58ccac3b4728a98b46838595d05e [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.712103 19981 tablet_replica.cc:333] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.712523 19981 raft_consensus.cc:2243] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.712854 19981 raft_consensus.cc:2272] T c55d8a42b8694ebf9b76fbb571a9814d P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.714293 19981 tablet_replica.cc:333] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.714758 19981 raft_consensus.cc:2243] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.715449 19981 raft_consensus.cc:2272] T cb9ad90d00ed4e249f54ebbe0b02e22e P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.716933 19981 tablet_replica.cc:333] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.717334 19981 raft_consensus.cc:2243] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.718047 19981 raft_consensus.cc:2272] T 770832072b40455b9a76ba3e907fb517 P 974a58ccac3b4728a98b46838595d05e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.719631 19981 tablet_replica.cc:333] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e: stopping tablet replica
I20251214 02:57:56.720023 19981 raft_consensus.cc:2243] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 5 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.720343 19981 raft_consensus.cc:2272] T fcdae6c6dc2445d591d0bbefed18e4b9 P 974a58ccac3b4728a98b46838595d05e [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.763024 19981 tablet_server.cc:195] TabletServer@127.19.131.66:0 shutdown complete.
I20251214 02:57:56.781741 19981 tablet_server.cc:178] TabletServer@127.19.131.67:0 shutting down...
I20251214 02:57:56.804922 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:57:56.805814 19981 tablet_replica.cc:333] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.806981 19981 raft_consensus.cc:2243] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.808156 19981 raft_consensus.cc:2272] T 4d39ca6708704cd48f2c17fd7fdd6ae7 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.810078 19981 tablet_replica.cc:333] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.810738 19981 raft_consensus.cc:2243] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.811280 19981 raft_consensus.cc:2272] T c411c9b2da7e4e2b83d854f004f13bfa P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.813017 19981 tablet_replica.cc:333] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.813530 19981 raft_consensus.cc:2243] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.813994 19981 raft_consensus.cc:2272] T 26f95721f9c4489892d51ed0aa794585 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.815722 19981 tablet_replica.cc:333] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.816267 19981 raft_consensus.cc:2243] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.817328 19981 raft_consensus.cc:2272] T bea937187ce744e99c7e48d4afc3469d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.819155 19981 tablet_replica.cc:333] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.819681 19981 raft_consensus.cc:2243] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.820152 19981 raft_consensus.cc:2272] T 99fd1f7626cf406db2c278a08948befb P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.821873 19981 tablet_replica.cc:333] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.822324 19981 raft_consensus.cc:2243] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.822888 19981 raft_consensus.cc:2272] T fcdae6c6dc2445d591d0bbefed18e4b9 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.824453 19981 tablet_replica.cc:333] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.824998 19981 raft_consensus.cc:2243] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:57:56.825941 19981 raft_consensus.cc:2272] T c55d8a42b8694ebf9b76fbb571a9814d P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.827749 19981 tablet_replica.cc:333] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.828158 19981 raft_consensus.cc:2243] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.828655 19981 raft_consensus.cc:2272] T cb9ad90d00ed4e249f54ebbe0b02e22e P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.830261 19981 tablet_replica.cc:333] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9: stopping tablet replica
I20251214 02:57:56.830691 19981 raft_consensus.cc:2243] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:57:56.831137 19981 raft_consensus.cc:2272] T 770832072b40455b9a76ba3e907fb517 P 4c28f10b4cbc48ab8f06564fc8a5edf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:56.857182 19981 tablet_server.cc:195] TabletServer@127.19.131.67:0 shutdown complete.
I20251214 02:57:56.875018 19981 master.cc:561] Master@127.19.131.126:44625 shutting down...
I20251214 02:57:57.072239 19981 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 1 LEADER]: Raft consensus shutting down.
I20251214 02:57:57.072911 19981 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:57:57.073254 19981 tablet_replica.cc:333] T 00000000000000000000000000000000 P c1368a111fd448f6a306ed9c107462fa: stopping tablet replica
I20251214 02:57:57.110184 19981 master.cc:583] Master@127.19.131.126:44625 shutdown complete.
4c28f10b4cbc48ab8f06564fc8a5edf9 1
9269ad98b10f4d61a2d9a943a117684d 4
974a58ccac3b4728a98b46838595d05e 4
4c28f10b4cbc48ab8f06564fc8a5edf9 3
9269ad98b10f4d61a2d9a943a117684d 3
974a58ccac3b4728a98b46838595d05e 3
4c28f10b4cbc48ab8f06564fc8a5edf9 1
9269ad98b10f4d61a2d9a943a117684d 0
974a58ccac3b4728a98b46838595d05e 8
4c28f10b4cbc48ab8f06564fc8a5edf9 3
9269ad98b10f4d61a2d9a943a117684d 3
974a58ccac3b4728a98b46838595d05e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30501 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20251214 02:57:57.150365 19981 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.131.126:33021
I20251214 02:57:57.151595 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:57.157353 20496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.157353 20497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.158080 20499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:57.158900 19981 server_base.cc:1047] running on GCE node
I20251214 02:57:57.159756 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:57.159961 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:57.160102 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681077160084 us; error 0 us; skew 500 ppm
I20251214 02:57:57.160615 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:57.163110 19981 webserver.cc:492] Webserver started at http://127.19.131.126:33035/ using document root <none> and password file <none>
I20251214 02:57:57.163655 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:57.163846 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:57.164103 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:57.165395 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/master-0-root/instance:
uuid: "5937feb5e09e47f7a7ce495e020158cd"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.170011 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251214 02:57:57.173540 20504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.174274 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251214 02:57:57.174628 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/master-0-root
uuid: "5937feb5e09e47f7a7ce495e020158cd"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.174919 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:57.191841 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:57.193092 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:57.230947 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.126:33021
I20251214 02:57:57.231027 20555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.126:33021 every 8 connection(s)
I20251214 02:57:57.234876 20556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.245168 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd: Bootstrap starting.
I20251214 02:57:57.250037 20556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:57.254360 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd: No bootstrap required, opened a new log
I20251214 02:57:57.256546 20556 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER }
I20251214 02:57:57.257022 20556 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:57.257256 20556 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5937feb5e09e47f7a7ce495e020158cd, State: Initialized, Role: FOLLOWER
I20251214 02:57:57.257786 20556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER }
I20251214 02:57:57.258239 20556 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 02:57:57.258462 20556 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 02:57:57.258781 20556 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:57.262774 20556 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER }
I20251214 02:57:57.263298 20556 leader_election.cc:304] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5937feb5e09e47f7a7ce495e020158cd; no voters:
I20251214 02:57:57.264446 20556 leader_election.cc:290] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 02:57:57.264855 20559 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:57.266117 20559 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 1 LEADER]: Becoming Leader. State: Replica: 5937feb5e09e47f7a7ce495e020158cd, State: Running, Role: LEADER
I20251214 02:57:57.266844 20559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER }
I20251214 02:57:57.267395 20556 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [sys.catalog]: configured and running, proceeding with master startup.
I20251214 02:57:57.272859 20561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5937feb5e09e47f7a7ce495e020158cd. Latest consensus state: current_term: 1 leader_uuid: "5937feb5e09e47f7a7ce495e020158cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER } }
I20251214 02:57:57.273514 20561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [sys.catalog]: This master's current role is: LEADER
I20251214 02:57:57.273991 20560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5937feb5e09e47f7a7ce495e020158cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5937feb5e09e47f7a7ce495e020158cd" member_type: VOTER } }
I20251214 02:57:57.274701 20560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [sys.catalog]: This master's current role is: LEADER
I20251214 02:57:57.276794 20567 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 02:57:57.282944 20567 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 02:57:57.283942 19981 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251214 02:57:57.291884 20567 catalog_manager.cc:1357] Generated new cluster ID: 9693ac14b5b54542a75d313d542ff208
I20251214 02:57:57.292220 20567 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 02:57:57.321687 20567 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 02:57:57.323006 20567 catalog_manager.cc:1514] Loading token signing keys...
I20251214 02:57:57.341274 20567 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd: Generated new TSK 0
I20251214 02:57:57.341917 20567 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 02:57:57.350176 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:57.355937 20577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.356734 20578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.358744 20580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:57.358856 19981 server_base.cc:1047] running on GCE node
I20251214 02:57:57.359805 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:57.359994 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:57.360148 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681077360131 us; error 0 us; skew 500 ppm
I20251214 02:57:57.360666 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:57.362932 19981 webserver.cc:492] Webserver started at http://127.19.131.65:45113/ using document root <none> and password file <none>
I20251214 02:57:57.363432 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:57.363612 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:57.363852 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:57.364913 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-0-root/instance:
uuid: "ca992896c00449d49ca22cdf754dbd40"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.369251 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251214 02:57:57.372362 20585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.373070 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 02:57:57.373350 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
uuid: "ca992896c00449d49ca22cdf754dbd40"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.373631 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:57.388557 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:57.389786 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:57.391284 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:57.393549 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:57.393738 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.393959 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:57.394114 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.432490 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.65:36613
I20251214 02:57:57.432591 20647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.65:36613 every 8 connection(s)
I20251214 02:57:57.436823 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:57.444613 20652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:57.447830 19981 server_base.cc:1047] running on GCE node
W20251214 02:57:57.448150 20653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.448648 20655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:57.450397 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:57:57.450640 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:57.450831 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681077450817 us; error 0 us; skew 500 ppm
I20251214 02:57:57.451445 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:57.452337 20648 heartbeater.cc:344] Connected to a master server at 127.19.131.126:33021
I20251214 02:57:57.452737 20648 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:57.453558 20648 heartbeater.cc:507] Master 127.19.131.126:33021 requested a full tablet report, sending...
I20251214 02:57:57.454082 19981 webserver.cc:492] Webserver started at http://127.19.131.66:36513/ using document root <none> and password file <none>
I20251214 02:57:57.454679 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:57.454857 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:57.455106 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:57.455737 20521 ts_manager.cc:194] Registered new tserver with Master: ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613)
I20251214 02:57:57.456236 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-1-root/instance:
uuid: "6b95b11145e4473e94b1313fd2eabf75"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.457408 20521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39546
I20251214 02:57:57.460954 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20251214 02:57:57.464375 20660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.465085 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 02:57:57.465363 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
uuid: "6b95b11145e4473e94b1313fd2eabf75"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.465621 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:57.489387 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:57.490630 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:57.492113 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:57.494314 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:57.494536 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.494783 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:57.494933 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.532639 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.66:46435
I20251214 02:57:57.532737 20722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.66:46435 every 8 connection(s)
I20251214 02:57:57.537432 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:57:57.545719 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.545717 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:57:57.547185 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:57:57.548519 20723 heartbeater.cc:344] Connected to a master server at 127.19.131.126:33021
I20251214 02:57:57.548763 19981 server_base.cc:1047] running on GCE node
I20251214 02:57:57.548882 20723 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:57.549628 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251214 02:57:57.549669 20723 heartbeater.cc:507] Master 127.19.131.126:33021 requested a full tablet report, sending...
W20251214 02:57:57.549856 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:57:57.550050 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681077550034 us; error 0 us; skew 500 ppm
I20251214 02:57:57.550618 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:57:57.551601 20521 ts_manager.cc:194] Registered new tserver with Master: 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:57.552994 19981 webserver.cc:492] Webserver started at http://127.19.131.67:36681/ using document root <none> and password file <none>
I20251214 02:57:57.553085 20521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39554
I20251214 02:57:57.553539 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:57:57.553757 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:57:57.554071 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:57:57.555222 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-2-root/instance:
uuid: "bba8a32c18e84d989607b148c701718b"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.559401 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20251214 02:57:57.562408 20734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.563174 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20251214 02:57:57.563441 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
uuid: "bba8a32c18e84d989607b148c701718b"
format_stamp: "Formatted at 2025-12-14 02:57:57 on dist-test-slave-8pjb"
I20251214 02:57:57.563727 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1765681046489641-19981-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:57:57.578799 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:57:57.579974 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:57:57.581358 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:57:57.583446 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:57:57.583647 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.583863 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:57:57.584021 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:57:57.622205 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.67:46875
I20251214 02:57:57.622311 20796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.67:46875 every 8 connection(s)
I20251214 02:57:57.634920 20797 heartbeater.cc:344] Connected to a master server at 127.19.131.126:33021
I20251214 02:57:57.635272 20797 heartbeater.cc:461] Registering TS with master...
I20251214 02:57:57.635938 20797 heartbeater.cc:507] Master 127.19.131.126:33021 requested a full tablet report, sending...
I20251214 02:57:57.637634 20521 ts_manager.cc:194] Registered new tserver with Master: bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:57.638646 19981 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013335668s
I20251214 02:57:57.638903 20521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39556
I20251214 02:57:57.641217 19981 test_util.cc:276] Using random seed: -1452580887
I20251214 02:57:57.662345 20521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39560:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 02:57:57.665241 20521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 02:57:57.737715 20762 tablet_service.cc:1505] Processing CreateTablet for tablet 1a942f14898c4cfaabc38f4ada704ead (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251214 02:57:57.739214 20762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a942f14898c4cfaabc38f4ada704ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.741298 20688 tablet_service.cc:1505] Processing CreateTablet for tablet 1a942f14898c4cfaabc38f4ada704ead (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251214 02:57:57.742652 20688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a942f14898c4cfaabc38f4ada704ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.744026 20759 tablet_service.cc:1505] Processing CreateTablet for tablet d50956b58a724d2b86316a213bfc6314 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251214 02:57:57.745152 20609 tablet_service.cc:1505] Processing CreateTablet for tablet e982dce2ff7648758cebe75226ebf7b5 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251214 02:57:57.745530 20613 tablet_service.cc:1505] Processing CreateTablet for tablet 1a942f14898c4cfaabc38f4ada704ead (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251214 02:57:57.746510 20609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e982dce2ff7648758cebe75226ebf7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.746919 20687 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2bdcdbe4244334b95ed4eabf856a39 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251214 02:57:57.747876 20612 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2bdcdbe4244334b95ed4eabf856a39 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251214 02:57:57.749186 20612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2bdcdbe4244334b95ed4eabf856a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.749075 20686 tablet_service.cc:1505] Processing CreateTablet for tablet b61e7d0235f247069c74b83373756b4a (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251214 02:57:57.749493 20607 tablet_service.cc:1505] Processing CreateTablet for tablet a29b17e78cf24d5987135f72e68b6822 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251214 02:57:57.750720 20607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29b17e78cf24d5987135f72e68b6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.751240 20685 tablet_service.cc:1505] Processing CreateTablet for tablet d50956b58a724d2b86316a213bfc6314 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251214 02:57:57.751770 20611 tablet_service.cc:1505] Processing CreateTablet for tablet b61e7d0235f247069c74b83373756b4a (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251214 02:57:57.753072 20611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61e7d0235f247069c74b83373756b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.745457 20759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50956b58a724d2b86316a213bfc6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.752988 20684 tablet_service.cc:1505] Processing CreateTablet for tablet e982dce2ff7648758cebe75226ebf7b5 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251214 02:57:57.755659 20608 tablet_service.cc:1505] Processing CreateTablet for tablet 0900b8afa00f4d819740e61db7e435f1 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251214 02:57:57.755955 20606 tablet_service.cc:1505] Processing CreateTablet for tablet d813f041d92b4a85a26ddd19f003e84f (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251214 02:57:57.757040 20761 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2bdcdbe4244334b95ed4eabf856a39 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251214 02:57:57.757793 20610 tablet_service.cc:1505] Processing CreateTablet for tablet d50956b58a724d2b86316a213bfc6314 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251214 02:57:57.757855 20760 tablet_service.cc:1505] Processing CreateTablet for tablet b61e7d0235f247069c74b83373756b4a (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251214 02:57:57.758414 20761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2bdcdbe4244334b95ed4eabf856a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.759243 20760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61e7d0235f247069c74b83373756b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.760708 20686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61e7d0235f247069c74b83373756b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.761090 20758 tablet_service.cc:1505] Processing CreateTablet for tablet e982dce2ff7648758cebe75226ebf7b5 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251214 02:57:57.761948 20687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2bdcdbe4244334b95ed4eabf856a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.762430 20758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e982dce2ff7648758cebe75226ebf7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.762554 20684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e982dce2ff7648758cebe75226ebf7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.763302 20613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a942f14898c4cfaabc38f4ada704ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.762250 20682 tablet_service.cc:1505] Processing CreateTablet for tablet a29b17e78cf24d5987135f72e68b6822 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251214 02:57:57.764289 20682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29b17e78cf24d5987135f72e68b6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.763990 20757 tablet_service.cc:1505] Processing CreateTablet for tablet 0900b8afa00f4d819740e61db7e435f1 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251214 02:57:57.761093 20756 tablet_service.cc:1505] Processing CreateTablet for tablet a29b17e78cf24d5987135f72e68b6822 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251214 02:57:57.765326 20757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0900b8afa00f4d819740e61db7e435f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.766775 20685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50956b58a724d2b86316a213bfc6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.766592 20683 tablet_service.cc:1505] Processing CreateTablet for tablet 0900b8afa00f4d819740e61db7e435f1 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251214 02:57:57.768467 20683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0900b8afa00f4d819740e61db7e435f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.766446 20606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d813f041d92b4a85a26ddd19f003e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.774883 20756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29b17e78cf24d5987135f72e68b6822. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.775851 20608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0900b8afa00f4d819740e61db7e435f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.777071 20610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50956b58a724d2b86316a213bfc6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.829872 20813 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:57.855065 20755 tablet_service.cc:1505] Processing CreateTablet for tablet d813f041d92b4a85a26ddd19f003e84f (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251214 02:57:57.856384 20755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d813f041d92b4a85a26ddd19f003e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.867952 20813 tablet_bootstrap.cc:654] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:57.875245 20812 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:57.881783 20608 tablet_service.cc:1505] Processing CreateTablet for tablet 948bafb5a8844d2cabfe1d2eb6c5d785 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251214 02:57:57.884022 20684 tablet_service.cc:1505] Processing CreateTablet for tablet 948bafb5a8844d2cabfe1d2eb6c5d785 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251214 02:57:57.886291 20684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948bafb5a8844d2cabfe1d2eb6c5d785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.887789 20756 tablet_service.cc:1505] Processing CreateTablet for tablet 948bafb5a8844d2cabfe1d2eb6c5d785 (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251214 02:57:57.889158 20756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948bafb5a8844d2cabfe1d2eb6c5d785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.900101 20608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948bafb5a8844d2cabfe1d2eb6c5d785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.906016 20759 tablet_service.cc:1505] Processing CreateTablet for tablet e08c401f218e44d6975b50cb229454eb (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251214 02:57:57.907281 20759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08c401f218e44d6975b50cb229454eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.911337 20609 tablet_service.cc:1505] Processing CreateTablet for tablet e08c401f218e44d6975b50cb229454eb (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251214 02:57:57.912539 20609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08c401f218e44d6975b50cb229454eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.884909 20688 tablet_service.cc:1505] Processing CreateTablet for tablet d813f041d92b4a85a26ddd19f003e84f (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251214 02:57:57.914604 20688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d813f041d92b4a85a26ddd19f003e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.921968 20686 tablet_service.cc:1505] Processing CreateTablet for tablet e08c401f218e44d6975b50cb229454eb (DEFAULT_TABLE table=test-workload [id=9f446dd5798b436bace0bb3c88ba69a5]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251214 02:57:57.923275 20686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08c401f218e44d6975b50cb229454eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:57:57.922370 20812 tablet_bootstrap.cc:654] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:57.934279 20813 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:57.946621 20813 ts_tablet_manager.cc:1403] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.117s user 0.015s sys 0.003s
I20251214 02:57:57.952461 20814 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:57.957182 20813 raft_consensus.cc:359] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:57.957898 20813 raft_consensus.cc:385] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:57.958199 20813 raft_consensus.cc:740] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:57.958591 20812 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:57.959029 20812 ts_tablet_manager.cc:1403] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.003s
I20251214 02:57:57.961763 20812 raft_consensus.cc:359] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:57.962443 20812 raft_consensus.cc:385] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:57.962752 20812 raft_consensus.cc:740] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:57.964241 20812 consensus_queue.cc:260] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:57.958875 20813 consensus_queue.cc:260] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:57.970350 20814 tablet_bootstrap.cc:654] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:57.972996 20648 heartbeater.cc:499] Master 127.19.131.126:33021 was elected leader, sending a full tablet report...
I20251214 02:57:57.973937 20813 ts_tablet_manager.cc:1434] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.025s user 0.009s sys 0.002s
I20251214 02:57:57.975029 20813 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:57.979423 20814 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:57.979880 20814 ts_tablet_manager.cc:1403] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.010s
I20251214 02:57:57.981565 20813 tablet_bootstrap.cc:654] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:57.982605 20814 raft_consensus.cc:359] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:57.983333 20814 raft_consensus.cc:385] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:57.983623 20814 raft_consensus.cc:740] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:57.984551 20814 consensus_queue.cc:260] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:57.991926 20723 heartbeater.cc:499] Master 127.19.131.126:33021 was elected leader, sending a full tablet report...
I20251214 02:57:57.992635 20812 ts_tablet_manager.cc:1434] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.033s user 0.013s sys 0.018s
I20251214 02:57:57.993705 20812 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.000308 20812 tablet_bootstrap.cc:654] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.012389 20812 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.012956 20812 ts_tablet_manager.cc:1403] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20251214 02:57:58.015908 20812 raft_consensus.cc:359] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.016705 20812 raft_consensus.cc:385] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.017000 20812 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.017926 20812 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.020392 20812 ts_tablet_manager.cc:1434] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20251214 02:57:58.021389 20812 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.027957 20812 tablet_bootstrap.cc:654] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.038590 20821 raft_consensus.cc:493] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.038612 20797 heartbeater.cc:499] Master 127.19.131.126:33021 was elected leader, sending a full tablet report...
I20251214 02:57:58.040261 20821 raft_consensus.cc:515] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.041478 20813 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.041940 20813 ts_tablet_manager.cc:1403] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.039s
I20251214 02:57:58.044768 20813 raft_consensus.cc:359] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.045460 20813 raft_consensus.cc:385] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.045742 20813 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.046407 20813 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.051002 20813 ts_tablet_manager.cc:1434] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20251214 02:57:58.051936 20813 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.039109 20812 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.039213 20814 ts_tablet_manager.cc:1434] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.059s user 0.025s sys 0.016s
I20251214 02:57:58.056948 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.057898 20813 tablet_bootstrap.cc:654] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.057962 20814 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
W20251214 02:57:58.058665 20737 leader_election.cc:343] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.056926 20812 ts_tablet_manager.cc:1403] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.008s
I20251214 02:57:58.062487 20812 raft_consensus.cc:359] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.063292 20812 raft_consensus.cc:385] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.063393 20821 leader_election.cc:290] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613)
I20251214 02:57:58.063665 20812 raft_consensus.cc:740] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.065135 20814 tablet_bootstrap.cc:654] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.068210 20812 consensus_queue.cc:260] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.070833 20812 ts_tablet_manager.cc:1434] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.011s user 0.001s sys 0.006s
I20251214 02:57:58.071821 20812 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.078155 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.078884 20698 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 0.
I20251214 02:57:58.079275 20813 tablet_bootstrap.cc:492] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.079738 20813 ts_tablet_manager.cc:1403] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.012s
I20251214 02:57:58.080420 20735 leader_election.cc:304] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters: ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:58.081203 20812 tablet_bootstrap.cc:654] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.082396 20813 raft_consensus.cc:359] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.083145 20813 raft_consensus.cc:385] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.083434 20813 raft_consensus.cc:740] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.084417 20814 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.084291 20813 consensus_queue.cc:260] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.084932 20814 ts_tablet_manager.cc:1403] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.009s
I20251214 02:57:58.085080 20821 raft_consensus.cc:2804] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.085458 20821 raft_consensus.cc:493] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.085793 20821 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.086985 20813 ts_tablet_manager.cc:1434] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251214 02:57:58.088064 20813 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.088068 20814 raft_consensus.cc:359] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.089076 20814 raft_consensus.cc:385] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.089417 20814 raft_consensus.cc:740] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.090221 20814 consensus_queue.cc:260] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.092469 20814 ts_tablet_manager.cc:1434] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 02:57:58.093354 20814 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.095176 20813 tablet_bootstrap.cc:654] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.098034 20812 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.098544 20812 ts_tablet_manager.cc:1403] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.003s
I20251214 02:57:58.100618 20814 tablet_bootstrap.cc:654] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.101302 20812 raft_consensus.cc:359] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.102070 20812 raft_consensus.cc:385] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.102381 20812 raft_consensus.cc:740] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.103240 20812 consensus_queue.cc:260] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.106191 20813 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.106675 20813 ts_tablet_manager.cc:1403] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20251214 02:57:58.107451 20817 raft_consensus.cc:493] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.107896 20817 raft_consensus.cc:515] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.109295 20813 raft_consensus.cc:359] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.109958 20813 raft_consensus.cc:385] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.110251 20813 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.111128 20813 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.113551 20814 tablet_bootstrap.cc:492] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.114053 20814 ts_tablet_manager.cc:1403] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20251214 02:57:58.115621 20813 ts_tablet_manager.cc:1434] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251214 02:57:58.116668 20813 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.116853 20817 leader_election.cc:290] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.117115 20814 raft_consensus.cc:359] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.117930 20814 raft_consensus.cc:385] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.118247 20814 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.119056 20814 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.121534 20814 ts_tablet_manager.cc:1434] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20251214 02:57:58.122507 20814 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.113759 20821 raft_consensus.cc:515] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.123554 20818 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.124006 20818 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.126494 20818 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.129988 20814 tablet_bootstrap.cc:654] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.131594 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:58.132193 20698 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.136130 20813 tablet_bootstrap.cc:654] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.135916 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
W20251214 02:57:58.137466 20661 leader_election.cc:343] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bba8a32c18e84d989607b148c701718b (127.19.131.67:46875): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.139200 20698 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 1.
I20251214 02:57:58.124010 20812 ts_tablet_manager.cc:1434] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20251214 02:57:58.143155 20735 leader_election.cc:304] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.147364 20814 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.147913 20814 ts_tablet_manager.cc:1403] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.005s
I20251214 02:57:58.150439 20814 raft_consensus.cc:359] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.151304 20814 raft_consensus.cc:385] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.151638 20814 raft_consensus.cc:740] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.152437 20814 consensus_queue.cc:260] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.145120 20812 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.154920 20823 raft_consensus.cc:2804] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.142556 20821 leader_election.cc:290] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613)
I20251214 02:57:58.173139 20812 tablet_bootstrap.cc:654] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.174374 20836 raft_consensus.cc:493] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.176504 20814 ts_tablet_manager.cc:1434] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.028s user 0.007s sys 0.006s
I20251214 02:57:58.176940 20836 raft_consensus.cc:515] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.179344 20814 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.180912 20823 raft_consensus.cc:697] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:58.181813 20823 consensus_queue.cc:237] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:58.184933 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.185599 20698 raft_consensus.cc:2468] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 0.
I20251214 02:57:58.186905 20735 leader_election.cc:304] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.184998 20817 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.187819 20817 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.189697 20817 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.190975 20837 raft_consensus.cc:2804] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.191423 20837 raft_consensus.cc:493] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.191679 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
W20251214 02:57:58.193197 20661 leader_election.cc:343] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bba8a32c18e84d989607b148c701718b (127.19.131.67:46875): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.191792 20836 leader_election.cc:290] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:58.192839 20519 catalog_manager.cc:5654] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 0 to 1, leader changed from <none> to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 1 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.193670 20837 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.196595 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:58.197158 20623 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.198485 20813 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.198994 20813 ts_tablet_manager.cc:1403] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.083s user 0.033s sys 0.026s
I20251214 02:57:58.201673 20813 raft_consensus.cc:359] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.202358 20813 raft_consensus.cc:385] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.202678 20813 raft_consensus.cc:740] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.203042 20814 tablet_bootstrap.cc:654] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.203342 20813 consensus_queue.cc:260] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.203903 20623 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 1.
I20251214 02:57:58.205999 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.207865 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.207712 20837 raft_consensus.cc:515] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.208438 20813 ts_tablet_manager.cc:1434] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251214 02:57:58.209077 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.209456 20813 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.209700 20622 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
W20251214 02:57:58.210693 20737 leader_election.cc:343] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.211545 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
W20251214 02:57:58.211942 20663 leader_election.cc:343] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.212061 20698 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.212425 20663 leader_election.cc:304] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75; no voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:58.214066 20663 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters: bba8a32c18e84d989607b148c701718b
I20251214 02:57:58.215925 20813 tablet_bootstrap.cc:654] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.215925 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
W20251214 02:57:58.217198 20737 leader_election.cc:343] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.218951 20698 raft_consensus.cc:2468] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 1.
I20251214 02:57:58.219498 20817 raft_consensus.cc:2749] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:58.221946 20817 raft_consensus.cc:2804] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.222266 20817 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.222786 20817 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.227556 20817 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.229015 20837 leader_election.cc:290] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:58.228869 20735 leader_election.cc:304] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters: ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:58.229209 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.229977 20823 raft_consensus.cc:2804] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.230472 20823 raft_consensus.cc:697] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:58.230506 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.231377 20698 raft_consensus.cc:2393] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca992896c00449d49ca22cdf754dbd40 in current term 1: Already voted for candidate 6b95b11145e4473e94b1313fd2eabf75 in this term.
I20251214 02:57:58.231783 20813 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.231453 20823 consensus_queue.cc:237] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.232232 20813 ts_tablet_manager.cc:1403] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.011s
I20251214 02:57:58.233453 20818 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.233950 20818 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.236025 20818 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.236831 20814 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.237365 20814 ts_tablet_manager.cc:1403] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.005s
I20251214 02:57:58.237805 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.237840 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.239514 20842 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:57:58.239907 20586 leader_election.cc:343] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bba8a32c18e84d989607b148c701718b (127.19.131.67:46875): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.240525 20586 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:57:58.239975 20842 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.240540 20814 raft_consensus.cc:359] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.242795 20842 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.243402 20818 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.243858 20817 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.244160 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.244791 20772 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.242946 20814 raft_consensus.cc:385] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.246703 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
W20251214 02:57:58.246345 20586 leader_election.cc:343] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.246784 20814 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.247664 20519 catalog_manager.cc:5654] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 0 to 1, leader changed from <none> to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 1 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:58.248407 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.248997 20814 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.247979 20661 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.250195 20818 raft_consensus.cc:2749] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 02:57:58.250335 20661 leader_election.cc:343] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bba8a32c18e84d989607b148c701718b (127.19.131.67:46875): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.247675 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.251039 20817 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.251183 20623 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.251427 20817 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.251958 20817 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.252199 20814 ts_tablet_manager.cc:1434] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20251214 02:57:58.253321 20814 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.257139 20622 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.259423 20814 tablet_bootstrap.cc:654] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.260694 20812 tablet_bootstrap.cc:492] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
W20251214 02:57:58.248003 20586 leader_election.cc:343] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bba8a32c18e84d989607b148c701718b (127.19.131.67:46875): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.261317 20586 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:57:58.260690 20813 raft_consensus.cc:359] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.262458 20813 raft_consensus.cc:385] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.262786 20813 raft_consensus.cc:740] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.261322 20812 ts_tablet_manager.cc:1403] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.014s
I20251214 02:57:58.263502 20813 consensus_queue.cc:260] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.266201 20663 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters: bba8a32c18e84d989607b148c701718b
I20251214 02:57:58.267475 20842 raft_consensus.cc:2804] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.268843 20842 raft_consensus.cc:697] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:58.269680 20842 consensus_queue.cc:237] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.271487 20818 raft_consensus.cc:2749] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:58.273403 20814 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.273852 20814 ts_tablet_manager.cc:1403] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20251214 02:57:58.274187 20813 ts_tablet_manager.cc:1434] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.042s user 0.007s sys 0.007s
I20251214 02:57:58.275203 20813 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.276788 20814 raft_consensus.cc:359] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.277540 20814 raft_consensus.cc:385] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.277822 20814 raft_consensus.cc:740] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.278507 20814 consensus_queue.cc:260] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.280928 20814 ts_tablet_manager.cc:1434] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251214 02:57:58.281812 20814 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.282040 20813 tablet_bootstrap.cc:654] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
W20251214 02:57:58.286659 20569 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:57:58.288085 20814 tablet_bootstrap.cc:654] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.291574 20813 tablet_bootstrap.cc:492] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.292050 20813 ts_tablet_manager.cc:1403] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20251214 02:57:58.292757 20518 catalog_manager.cc:5654] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.294847 20813 raft_consensus.cc:359] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.295528 20813 raft_consensus.cc:385] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.295816 20813 raft_consensus.cc:740] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.296468 20813 consensus_queue.cc:260] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.298705 20813 ts_tablet_manager.cc:1434] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20251214 02:57:58.299611 20813 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.301446 20847 raft_consensus.cc:493] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.301946 20847 raft_consensus.cc:515] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.304383 20847 leader_election.cc:290] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.305657 20813 tablet_bootstrap.cc:654] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.306123 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.312623 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.313344 20772 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 0.
I20251214 02:57:58.315271 20586 leader_election.cc:304] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
W20251214 02:57:58.317572 20586 leader_election.cc:343] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:57:58.318365 20818 raft_consensus.cc:2804] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.318719 20818 raft_consensus.cc:493] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.319042 20818 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.320122 20814 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.320592 20814 ts_tablet_manager.cc:1403] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.016s
I20251214 02:57:58.322670 20817 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.323290 20814 raft_consensus.cc:359] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.323944 20814 raft_consensus.cc:385] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.324249 20814 raft_consensus.cc:740] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.325034 20814 consensus_queue.cc:260] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.325654 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:58.325748 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.326344 20622 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.326372 20772 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.330284 20814 ts_tablet_manager.cc:1434] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20251214 02:57:58.331288 20814 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.333501 20772 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.334667 20661 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.316291 20812 raft_consensus.cc:359] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.335436 20842 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.335780 20817 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.335909 20842 raft_consensus.cc:697] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:58.337113 20814 tablet_bootstrap.cc:654] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.336740 20842 consensus_queue.cc:237] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.335513 20812 raft_consensus.cc:385] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.338644 20812 raft_consensus.cc:740] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.339425 20812 consensus_queue.cc:260] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.341818 20812 ts_tablet_manager.cc:1434] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.079s user 0.005s sys 0.001s
I20251214 02:57:58.343181 20812 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.346529 20622 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.347657 20818 raft_consensus.cc:515] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.353811 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:58.354418 20698 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.356324 20520 catalog_manager.cc:5654] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.361515 20698 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.362875 20586 leader_election.cc:304] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.363708 20847 raft_consensus.cc:2804] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.370847 20818 leader_election.cc:290] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.371817 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.372455 20772 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.377027 20813 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.377592 20813 ts_tablet_manager.cc:1403] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.078s user 0.014s sys 0.043s
I20251214 02:57:58.378031 20812 tablet_bootstrap.cc:654] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.380142 20813 raft_consensus.cc:359] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.380865 20813 raft_consensus.cc:385] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.381201 20813 raft_consensus.cc:740] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.382134 20847 raft_consensus.cc:697] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:58.382006 20813 consensus_queue.cc:260] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.382774 20772 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.383028 20847 consensus_queue.cc:237] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.385330 20813 ts_tablet_manager.cc:1434] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 02:57:58.386309 20813 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.389016 20814 tablet_bootstrap.cc:492] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.389463 20814 ts_tablet_manager.cc:1403] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.058s user 0.031s sys 0.014s
I20251214 02:57:58.392174 20812 tablet_bootstrap.cc:492] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.392009 20814 raft_consensus.cc:359] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.392654 20813 tablet_bootstrap.cc:654] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.392689 20814 raft_consensus.cc:385] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.392789 20812 ts_tablet_manager.cc:1403] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.000s
I20251214 02:57:58.393110 20814 raft_consensus.cc:740] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.392429 20520 catalog_manager.cc:5654] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.393905 20814 consensus_queue.cc:260] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.396059 20812 raft_consensus.cc:359] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.396822 20812 raft_consensus.cc:385] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.398044 20812 raft_consensus.cc:740] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.398911 20812 consensus_queue.cc:260] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.398351 20814 ts_tablet_manager.cc:1434] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251214 02:57:58.403287 20814 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.409029 20814 tablet_bootstrap.cc:654] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.413220 20812 ts_tablet_manager.cc:1434] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20251214 02:57:58.416805 20812 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.422209 20812 tablet_bootstrap.cc:654] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.424417 20813 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.424822 20813 ts_tablet_manager.cc:1403] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.013s
I20251214 02:57:58.427206 20814 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.426959 20813 raft_consensus.cc:359] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.427636 20813 raft_consensus.cc:385] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.427630 20814 ts_tablet_manager.cc:1403] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20251214 02:57:58.428045 20813 raft_consensus.cc:740] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.428786 20813 consensus_queue.cc:260] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.430186 20814 raft_consensus.cc:359] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.430665 20814 raft_consensus.cc:385] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.430812 20813 ts_tablet_manager.cc:1434] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251214 02:57:58.430831 20814 raft_consensus.cc:740] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.429380 20812 tablet_bootstrap.cc:492] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.431795 20813 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: Bootstrap starting.
I20251214 02:57:58.431573 20814 consensus_queue.cc:260] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.431955 20812 ts_tablet_manager.cc:1403] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.001s
I20251214 02:57:58.434787 20812 raft_consensus.cc:359] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.435500 20812 raft_consensus.cc:385] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.435822 20812 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.436152 20814 ts_tablet_manager.cc:1434] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20251214 02:57:58.437024 20814 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: Bootstrap starting.
I20251214 02:57:58.436651 20812 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.437502 20813 tablet_bootstrap.cc:654] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.442776 20814 tablet_bootstrap.cc:654] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.446002 20812 ts_tablet_manager.cc:1434] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251214 02:57:58.449611 20813 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: No bootstrap required, opened a new log
I20251214 02:57:58.450064 20813 ts_tablet_manager.cc:1403] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20251214 02:57:58.452549 20813 raft_consensus.cc:359] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.453202 20813 raft_consensus.cc:385] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.453367 20814 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: No bootstrap required, opened a new log
I20251214 02:57:58.453473 20813 raft_consensus.cc:740] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.453873 20814 ts_tablet_manager.cc:1403] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
I20251214 02:57:58.454171 20813 consensus_queue.cc:260] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.454793 20812 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.456248 20813 ts_tablet_manager.cc:1434] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251214 02:57:58.456346 20814 raft_consensus.cc:359] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.457002 20814 raft_consensus.cc:385] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.457252 20814 raft_consensus.cc:740] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.457899 20814 consensus_queue.cc:260] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.459324 20855 raft_consensus.cc:493] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.459777 20855 raft_consensus.cc:515] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.461300 20812 tablet_bootstrap.cc:654] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.461724 20855 leader_election.cc:290] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.463198 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.463455 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.463791 20698 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 0.
I20251214 02:57:58.464071 20772 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 0.
I20251214 02:57:58.464453 20814 ts_tablet_manager.cc:1434] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20251214 02:57:58.464962 20586 leader_election.cc:304] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.465647 20855 raft_consensus.cc:2804] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.466004 20855 raft_consensus.cc:493] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.466324 20855 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.472957 20812 tablet_bootstrap.cc:492] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.472960 20855 raft_consensus.cc:515] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.473582 20812 ts_tablet_manager.cc:1403] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20251214 02:57:58.474987 20855 leader_election.cc:290] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.475672 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:58.475867 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.476131 20698 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.476358 20772 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.480811 20698 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.480825 20772 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.481782 20586 leader_election.cc:304] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.482120 20812 raft_consensus.cc:359] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.482450 20855 raft_consensus.cc:2804] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.482656 20812 raft_consensus.cc:385] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.482962 20812 raft_consensus.cc:740] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.482967 20855 raft_consensus.cc:697] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:58.483655 20812 consensus_queue.cc:260] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.483839 20855 consensus_queue.cc:237] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.485631 20812 ts_tablet_manager.cc:1434] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251214 02:57:58.486495 20812 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.491285 20812 tablet_bootstrap.cc:654] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.492558 20518 catalog_manager.cc:5654] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.497002 20812 tablet_bootstrap.cc:492] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.497445 20812 ts_tablet_manager.cc:1403] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20251214 02:57:58.499411 20812 raft_consensus.cc:359] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.499846 20812 raft_consensus.cc:385] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.500061 20812 raft_consensus.cc:740] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.500587 20812 consensus_queue.cc:260] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.501981 20812 ts_tablet_manager.cc:1434] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 02:57:58.502749 20812 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: Bootstrap starting.
I20251214 02:57:58.505856 20842 raft_consensus.cc:493] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.506235 20842 raft_consensus.cc:515] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.506780 20812 tablet_bootstrap.cc:654] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: Neither blocks nor log segments found. Creating new log.
I20251214 02:57:58.507180 20823 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.507606 20823 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.507943 20842 leader_election.cc:290] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.509454 20823 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:58.510130 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.510726 20622 raft_consensus.cc:2468] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.511303 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.511627 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.512027 20623 raft_consensus.cc:2393] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bba8a32c18e84d989607b148c701718b in current term 1: Already voted for candidate 6b95b11145e4473e94b1313fd2eabf75 in this term.
I20251214 02:57:58.512193 20772 raft_consensus.cc:2468] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.513293 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.513314 20661 leader_election.cc:304] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.514544 20842 raft_consensus.cc:2804] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.514765 20735 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bba8a32c18e84d989607b148c701718b; no voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:58.514974 20842 raft_consensus.cc:493] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.515415 20842 raft_consensus.cc:3060] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.515765 20812 tablet_bootstrap.cc:492] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: No bootstrap required, opened a new log
I20251214 02:57:58.515895 20823 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.516150 20812 ts_tablet_manager.cc:1403] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20251214 02:57:58.518199 20812 raft_consensus.cc:359] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.518669 20812 raft_consensus.cc:385] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:57:58.518865 20812 raft_consensus.cc:740] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Initialized, Role: FOLLOWER
I20251214 02:57:58.519393 20812 consensus_queue.cc:260] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.521184 20812 ts_tablet_manager.cc:1434] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20251214 02:57:58.522446 20823 raft_consensus.cc:2749] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:58.522454 20842 raft_consensus.cc:515] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.524211 20842 leader_election.cc:290] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.525074 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:58.525269 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.525542 20623 raft_consensus.cc:3060] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.525813 20772 raft_consensus.cc:3060] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.530794 20623 raft_consensus.cc:2468] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.531086 20772 raft_consensus.cc:2468] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.532001 20663 leader_election.cc:304] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.532740 20842 raft_consensus.cc:2804] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.533114 20842 raft_consensus.cc:697] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:58.533701 20842 consensus_queue.cc:237] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.536579 20842 raft_consensus.cc:493] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.536971 20842 raft_consensus.cc:515] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.539188 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.539852 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.540403 20623 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.540493 20772 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 0.
I20251214 02:57:58.541610 20663 leader_election.cc:304] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.541895 20518 catalog_manager.cc:5654] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.542444 20817 raft_consensus.cc:2804] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.542872 20817 raft_consensus.cc:493] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.543167 20817 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.539664 20842 leader_election.cc:290] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.548781 20817 raft_consensus.cc:515] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.550518 20817 leader_election.cc:290] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.550962 20855 raft_consensus.cc:493] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.551322 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:58.551538 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.551419 20855 raft_consensus.cc:515] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.551899 20623 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.552050 20772 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.553299 20855 leader_election.cc:290] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.554499 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.555256 20698 raft_consensus.cc:2393] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca992896c00449d49ca22cdf754dbd40 in current term 1: Already voted for candidate bba8a32c18e84d989607b148c701718b in this term.
I20251214 02:57:58.556206 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.557446 20772 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.557643 20586 leader_election.cc:304] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:57:58.558032 20623 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 1.
I20251214 02:57:58.558425 20661 leader_election.cc:304] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.558547 20855 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.559198 20817 raft_consensus.cc:2804] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.559571 20817 raft_consensus.cc:697] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:58.560293 20817 consensus_queue.cc:237] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.563830 20855 raft_consensus.cc:2749] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:57:58.567247 20518 catalog_manager.cc:5654] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.643105 20823 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.649176 20823 consensus_queue.cc:1048] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.656466 20823 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.667793 20823 consensus_queue.cc:1048] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.682134 20821 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.682538 20821 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.684196 20821 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:58.685035 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:57:58.685182 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.685752 20698 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 0.
I20251214 02:57:58.686494 20622 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 0.
I20251214 02:57:58.686707 20735 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:58.687489 20821 raft_consensus.cc:2804] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.687834 20821 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.688071 20821 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.693001 20821 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.695473 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:58.695665 20821 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:58.695921 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:58.695971 20622 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.696547 20698 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.700716 20622 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 1.
I20251214 02:57:58.700832 20698 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 1.
I20251214 02:57:58.701638 20737 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.702360 20821 raft_consensus.cc:2804] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.702737 20821 raft_consensus.cc:697] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:58.703415 20821 consensus_queue.cc:237] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.709884 20518 catalog_manager.cc:5654] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 0 to 1, leader changed from <none> to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 1 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:58.728966 20817 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.738533 20842 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.791697 20855 raft_consensus.cc:493] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:57:58.792167 20855 raft_consensus.cc:515] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.794016 20855 leader_election.cc:290] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.795019 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:57:58.795302 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:57:58.795701 20698 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 0.
I20251214 02:57:58.795859 20772 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 0.
I20251214 02:57:58.796742 20586 leader_election.cc:304] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.797410 20855 raft_consensus.cc:2804] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:57:58.797685 20855 raft_consensus.cc:493] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:57:58.797935 20855 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.802403 20855 raft_consensus.cc:515] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.803997 20855 leader_election.cc:290] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:58.804832 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:58.804982 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:58.805318 20698 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.805472 20772 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:57:58.810087 20772 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.810078 20698 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 1.
I20251214 02:57:58.811127 20586 leader_election.cc:304] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:58.811801 20855 raft_consensus.cc:2804] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:57:58.812179 20855 raft_consensus.cc:697] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:58.812865 20855 consensus_queue.cc:237] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:58.820252 20518 catalog_manager.cc:5654] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:58.844870 20855 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.849094 20842 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.855865 20882 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:57:58.870000 20817 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.905169 20882 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:58.915263 20847 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.004101 20817 consensus_queue.cc:1048] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.013661 20848 consensus_queue.cc:1048] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.084702 20848 consensus_queue.cc:1048] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.097939 20817 consensus_queue.cc:1048] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.102273 20772 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:55812
I20251214 02:57:59.102957 20772 raft_consensus.cc:606] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:59.119087 20698 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:59782
I20251214 02:57:59.119558 20698 raft_consensus.cc:606] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:59.123716 20772 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:55812
I20251214 02:57:59.124191 20772 raft_consensus.cc:606] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:59.128293 20698 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:59782
I20251214 02:57:59.128726 20698 raft_consensus.cc:606] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:57:59.132632 20772 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
mode: GRACEFUL
new_leader_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:55812
I20251214 02:57:59.133111 20772 raft_consensus.cc:606] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Received request to transfer leadership to TS 6b95b11145e4473e94b1313fd2eabf75
I20251214 02:57:59.148329 20622 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:52208
I20251214 02:57:59.148799 20622 raft_consensus.cc:606] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Received request to transfer leadership to TS 6b95b11145e4473e94b1313fd2eabf75
I20251214 02:57:59.157766 20622 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:52208
I20251214 02:57:59.158221 20622 raft_consensus.cc:606] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Received request to transfer leadership to TS bba8a32c18e84d989607b148c701718b
I20251214 02:57:59.162839 20821 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.167747 20848 raft_consensus.cc:993] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.168176 20817 raft_consensus.cc:1081] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.170118 20622 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52206
I20251214 02:57:59.170765 20622 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.171131 20622 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.175740 20823 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.177911 20622 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.179625 20622 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.180828 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.181391 20697 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:59.181958 20697 raft_consensus.cc:3055] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.182256 20697 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:59.183102 20697 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.184360 20697 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.181409 20772 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.185627 20823 raft_consensus.cc:993] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: : Instructing follower 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:57:59.186084 20821 raft_consensus.cc:1081] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Signalling peer 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:57:59.190636 20823 raft_consensus.cc:993] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.191071 20823 raft_consensus.cc:1081] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.191769 20772 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.192662 20698 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:59770
I20251214 02:57:59.192934 20622 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52196
I20251214 02:57:59.193254 20698 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.193473 20622 raft_consensus.cc:493] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.193645 20698 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.193848 20622 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.192961 20586 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:59.200245 20622 raft_consensus.cc:515] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:59.200241 20698 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.200958 20847 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.201527 20847 raft_consensus.cc:697] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.202472 20697 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.202307 20847 consensus_queue.cc:237] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.203379 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:59.204023 20696 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.205250 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.205786 20772 raft_consensus.cc:3055] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.206010 20622 leader_election.cc:290] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.206104 20772 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:59.205776 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.207134 20772 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.207829 20771 raft_consensus.cc:3055] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.208163 20771 raft_consensus.cc:740] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:59.208232 20772 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.208899 20771 consensus_queue.cc:260] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:59.209965 20771 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.206094 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:59.210841 20696 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.211764 20623 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.211265 20518 catalog_manager.cc:5654] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 1 to 2, leader changed from 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:59.211980 20586 leader_election.cc:304] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:59.215343 20847 raft_consensus.cc:2804] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.216049 20847 raft_consensus.cc:697] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.216615 20771 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.216915 20847 consensus_queue.cc:237] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:57:59.203375 20698 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 2 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.214005 20772 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 2.
I20251214 02:57:59.219416 20661 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:59.220258 20817 raft_consensus.cc:2804] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.220700 20817 raft_consensus.cc:697] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:59.221546 20623 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 2.
I20251214 02:57:59.221436 20817 consensus_queue.cc:237] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.228266 20520 catalog_manager.cc:5654] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 1 to 2, leader changed from bba8a32c18e84d989607b148c701718b (127.19.131.67) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:59.228216 20518 catalog_manager.cc:5654] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 1 to 2, leader changed from bba8a32c18e84d989607b148c701718b (127.19.131.67) to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:59.281728 20848 raft_consensus.cc:993] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.282163 20817 raft_consensus.cc:1081] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.283516 20623 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52206
I20251214 02:57:59.284013 20623 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.284261 20623 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.288568 20623 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.289991 20623 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.290880 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:59.291205 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.291450 20698 raft_consensus.cc:3055] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.291714 20772 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.291791 20698 raft_consensus.cc:740] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:59.292582 20698 consensus_queue.cc:260] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.293797 20698 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.298486 20772 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.299748 20586 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:59.300493 20698 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.300654 20847 raft_consensus.cc:2804] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.301170 20847 raft_consensus.cc:697] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.301973 20847 consensus_queue.cc:237] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.308032 20520 catalog_manager.cc:5654] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 1 to 2, leader changed from 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:59.315590 20882 raft_consensus.cc:993] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: : Instructing follower 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:57:59.315984 20847 raft_consensus.cc:1081] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Signalling peer 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:57:59.317477 20698 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:59772
I20251214 02:57:59.318040 20698 raft_consensus.cc:493] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.318320 20698 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.324376 20698 raft_consensus.cc:515] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.325903 20698 leader_election.cc:290] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 2 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.326658 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:59.326859 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.327342 20772 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.327207 20623 raft_consensus.cc:3055] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.327785 20623 raft_consensus.cc:740] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.328405 20623 consensus_queue.cc:260] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.329272 20623 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.331877 20772 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 2.
I20251214 02:57:59.332773 20661 leader_election.cc:304] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:57:59.333369 20817 raft_consensus.cc:2804] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.333711 20817 raft_consensus.cc:697] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:57:59.333861 20623 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 2.
I20251214 02:57:59.334436 20817 consensus_queue.cc:237] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.340050 20520 catalog_manager.cc:5654] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 1 to 2, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:59.361097 20882 consensus_queue.cc:1048] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.367471 20847 raft_consensus.cc:993] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: : Instructing follower bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:57:59.368049 20818 raft_consensus.cc:1081] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Signalling peer bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:57:59.369565 20771 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:55796
I20251214 02:57:59.370154 20771 raft_consensus.cc:493] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.371239 20771 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.372880 20818 consensus_queue.cc:1048] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:57:59.380496 20771 raft_consensus.cc:515] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.384874 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:57:59.385378 20623 raft_consensus.cc:3055] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.385641 20623 raft_consensus.cc:740] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.386386 20623 consensus_queue.cc:260] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.387374 20623 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.392002 20623 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 2.
I20251214 02:57:59.392758 20771 leader_election.cc:290] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 2 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:57:59.392884 20737 leader_election.cc:304] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:59.393759 20836 raft_consensus.cc:2804] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.393788 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:59.394299 20836 raft_consensus.cc:697] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:59.394361 20698 raft_consensus.cc:3060] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.395131 20836 consensus_queue.cc:237] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.399652 20698 raft_consensus.cc:2468] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 2.
I20251214 02:57:59.400795 20520 catalog_manager.cc:5654] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 1 to 2, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 2 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:57:59.544193 20821 raft_consensus.cc:993] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.544603 20836 raft_consensus.cc:1081] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:57:59.545940 20623 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52196
I20251214 02:57:59.546468 20623 raft_consensus.cc:493] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:57:59.546787 20623 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.551319 20623 raft_consensus.cc:515] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.552767 20623 leader_election.cc:290] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:57:59.553650 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:57:59.553874 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d50956b58a724d2b86316a213bfc6314" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:57:59.554224 20698 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.554550 20771 raft_consensus.cc:3055] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:57:59.554916 20771 raft_consensus.cc:740] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:57:59.555735 20771 consensus_queue.cc:260] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.556955 20771 raft_consensus.cc:3060] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:57:59.560854 20698 raft_consensus.cc:2468] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.561964 20586 leader_election.cc:304] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:57:59.562778 20882 raft_consensus.cc:2804] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:57:59.563141 20882 raft_consensus.cc:697] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:57:59.563215 20771 raft_consensus.cc:2468] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:57:59.563928 20882 consensus_queue.cc:237] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:57:59.569887 20520 catalog_manager.cc:5654] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 1 to 2, leader changed from bba8a32c18e84d989607b148c701718b (127.19.131.67) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:57:59.612200 20771 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.613252 20882 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.620165 20698 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.621629 20847 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.627511 20771 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.628597 20817 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.635975 20623 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.637179 20842 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.693653 20698 raft_consensus.cc:1275] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.694921 20847 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.702433 20771 raft_consensus.cc:1275] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.703604 20696 raft_consensus.cc:1275] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.704707 20818 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.705003 20882 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.718889 20771 raft_consensus.cc:1275] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.720554 20847 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.735694 20771 raft_consensus.cc:1275] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.736757 20817 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.743572 20623 raft_consensus.cc:1275] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.744958 20848 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.885399 20623 raft_consensus.cc:1275] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.886520 20836 consensus_queue.cc:1048] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.893301 20698 raft_consensus.cc:1275] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.894704 20902 consensus_queue.cc:1048] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:57:59.991348 20698 raft_consensus.cc:1275] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:57:59.992401 20847 consensus_queue.cc:1048] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:00.001370 20771 raft_consensus.cc:1275] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:00.002669 20818 consensus_queue.cc:1048] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:02.187505 19981 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20251214 02:58:02.188148 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:02.203675 20623 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:52224
I20251214 02:58:02.204171 20623 raft_consensus.cc:606] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Received request to transfer leadership to TS bba8a32c18e84d989607b148c701718b
I20251214 02:58:02.206243 20623 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:52224
I20251214 02:58:02.206694 20623 raft_consensus.cc:606] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Received request to transfer leadership to TS bba8a32c18e84d989607b148c701718b
I20251214 02:58:02.207571 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251214 02:58:02.208003 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:02.351466 20818 raft_consensus.cc:993] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: : Instructing follower bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:02.351938 20908 raft_consensus.cc:1081] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Signalling peer bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:02.353331 20771 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:55796
I20251214 02:58:02.353804 20771 raft_consensus.cc:493] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:02.354087 20771 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:02.358721 20771 raft_consensus.cc:515] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.360473 20771 leader_election.cc:290] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 2 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:02.361392 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:02.361543 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:02.362028 20623 raft_consensus.cc:3055] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:02.362087 20698 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:02.362296 20623 raft_consensus.cc:740] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:02.362975 20623 consensus_queue.cc:260] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.363867 20623 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:02.367240 20698 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 2.
I20251214 02:58:02.368104 20735 leader_election.cc:304] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:58:02.368850 20902 raft_consensus.cc:2804] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:02.369467 20623 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 2.
I20251214 02:58:02.370052 20902 raft_consensus.cc:697] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:02.370939 20902 consensus_queue.cc:237] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.377887 20519 catalog_manager.cc:5654] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 1 to 2, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 2 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:02.472435 20818 raft_consensus.cc:993] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: : Instructing follower bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:02.472846 20908 raft_consensus.cc:1081] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Signalling peer bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:02.474165 20771 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:55796
I20251214 02:58:02.474639 20771 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:02.474915 20771 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:02.479688 20771 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.481092 20771 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:02.481920 20623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:02.482120 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:02.482393 20623 raft_consensus.cc:3055] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:02.482627 20698 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:02.482733 20623 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:02.483400 20623 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.484249 20623 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:02.488085 20698 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 3.
I20251214 02:58:02.488658 20623 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 3.
I20251214 02:58:02.488927 20735 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:58:02.489760 20902 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:02.490201 20902 raft_consensus.cc:697] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:02.490980 20902 consensus_queue.cc:237] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:02.497236 20519 catalog_manager.cc:5654] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 2 to 3, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 3 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:02.837173 20623 raft_consensus.cc:1275] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:02.837174 20698 raft_consensus.cc:1275] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:02.838443 20909 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:02.839089 20902 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:02.926488 20698 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:02.927609 20909 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:02.935740 20623 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:02.937232 20909 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:03.208900 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:03.214246 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:03.214669 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:04.215581 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:04.221303 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:04.221801 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:05.222739 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:05.228389 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:05.228825 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:06.229744 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:06.235247 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:06.235674 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:07.236692 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:07.242897 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:07.243326 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:08.244357 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:08.249413 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:08.249785 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:09.250626 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:09.255625 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:09.255991 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:10.256762 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:10.261765 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:10.262130 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:11.276561 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:11.290652 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:11.297533 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:12.136534 20934 raft_consensus.cc:493] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b95b11145e4473e94b1313fd2eabf75)
I20251214 02:58:12.137391 20934 raft_consensus.cc:515] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.140379 20934 leader_election.cc:290] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.164534 20697 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
W20251214 02:58:12.266235 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08000541e0 after lost signal to thread 19984
I20251214 02:58:12.294514 20936 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6b95b11145e4473e94b1313fd2eabf75)
I20251214 02:58:12.295280 20936 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.302623 20936 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:12.304718 20934 raft_consensus.cc:493] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b95b11145e4473e94b1313fd2eabf75)
I20251214 02:58:12.305250 20934 raft_consensus.cc:515] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.307260 20934 leader_election.cc:290] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.308912 20697 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
W20251214 02:58:12.322985 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08000548c0 after lost signal to thread 20555
I20251214 02:58:12.337215 20934 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bba8a32c18e84d989607b148c701718b)
I20251214 02:58:12.338083 20937 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bba8a32c18e84d989607b148c701718b)
I20251214 02:58:12.337920 20934 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.338969 20937 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.339726 19981 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20251214 02:58:12.340618 20934 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.341169 20937 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.341969 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:58:12.344978 20696 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.346556 20770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "948bafb5a8844d2cabfe1d2eb6c5d785" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:58:12.346751 20586 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:12.347544 20934 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:58:12.347919 20934 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bba8a32c18e84d989607b148c701718b)
I20251214 02:58:12.348274 20934 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:12.349881 20770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:58:12.350193 20772 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:55812
I20251214 02:58:12.350932 20772 raft_consensus.cc:606] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.351800 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:58:12.342790 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
W20251214 02:58:12.356709 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08000ef4a0 after lost signal to thread 20647
W20251214 02:58:12.360147 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08001b66e0 after lost signal to thread 20722
W20251214 02:58:12.361341 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7a00 after lost signal to thread 20796
I20251214 02:58:12.367105 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:58:12.364751 20934 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.369673 20937 raft_consensus.cc:493] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bba8a32c18e84d989607b148c701718b)
I20251214 02:58:12.369949 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:58:12.370208 20937 raft_consensus.cc:515] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.371214 20622 raft_consensus.cc:2393] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b95b11145e4473e94b1313fd2eabf75 in current term 4: Already voted for candidate ca992896c00449d49ca22cdf754dbd40 in this term.
I20251214 02:58:12.371917 20934 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 4 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.372423 20937 leader_election.cc:290] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.377576 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b" is_pre_election: true
I20251214 02:58:12.389091 20771 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:55812
I20251214 02:58:12.389685 20771 raft_consensus.cc:606] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.391469 20586 leader_election.cc:304] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:58:12.392477 20941 raft_consensus.cc:2749] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:12.393193 20586 leader_election.cc:304] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:58:12.394932 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:12.395067 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:12.395277 20663 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75; no voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.396464 20698 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:59782
I20251214 02:58:12.397023 20698 raft_consensus.cc:606] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.397459 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e08c401f218e44d6975b50cb229454eb" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:58:12.398022 20939 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:12.398649 20941 raft_consensus.cc:2749] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:58:12.400547 20622 raft_consensus.cc:1240] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Rejecting Update request from peer bba8a32c18e84d989607b148c701718b for earlier term 3. Current term is 4. Ops: []
I20251214 02:58:12.402899 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:58:12.403697 20698 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:59782
I20251214 02:58:12.404212 20698 raft_consensus.cc:606] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.406499 20939 raft_consensus.cc:2749] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:58:12.406767 20663 leader_election.cc:304] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75; no voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.407727 20939 raft_consensus.cc:2749] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:58:12.409911 20937 raft_consensus.cc:993] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.409808 20935 consensus_queue.cc:1059] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:58:12.410621 20937 raft_consensus.cc:1081] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.410917 20935 raft_consensus.cc:3055] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:58:12.411264 20935 raft_consensus.cc:740] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:12.411479 20698 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
mode: GRACEFUL
new_leader_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:59782
I20251214 02:58:12.411947 20698 raft_consensus.cc:606] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Received request to transfer leadership to TS ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.412053 20935 consensus_queue.cc:260] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.413511 20935 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:12.419476 20586 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca992896c00449d49ca22cdf754dbd40; no voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b
I20251214 02:58:12.420413 20941 raft_consensus.cc:2749] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:58:12.420737 20925 raft_consensus.cc:993] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.419571 20737 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bba8a32c18e84d989607b148c701718b; no voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40
I20251214 02:58:12.421178 20622 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52206
I20251214 02:58:12.421492 20936 raft_consensus.cc:2749] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:58:12.421200 20925 raft_consensus.cc:1081] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.422385 20622 raft_consensus.cc:493] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:12.422758 20622 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.423493 20621 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52196
I20251214 02:58:12.424021 20621 raft_consensus.cc:493] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:12.424372 20621 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.429047 20622 raft_consensus.cc:515] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.430944 20622 leader_election.cc:290] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.431278 20621 raft_consensus.cc:515] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.431676 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:12.432435 20698 raft_consensus.cc:3055] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:12.432763 20698 raft_consensus.cc:740] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:58:12.432731 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a29b17e78cf24d5987135f72e68b6822" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:12.433357 20621 leader_election.cc:290] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.433533 20771 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.433815 20698 consensus_queue.cc:260] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.434909 20698 raft_consensus.cc:3060] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.437539 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:12.438098 20696 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.440052 20771 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.445070 20696 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.447288 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d2bdcdbe4244334b95ed4eabf856a39" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:12.447929 20771 raft_consensus.cc:3055] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:12.448262 20771 raft_consensus.cc:740] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:12.448414 20698 raft_consensus.cc:2468] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.449048 20771 consensus_queue.cc:260] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.450147 20771 raft_consensus.cc:3060] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.451503 20586 leader_election.cc:304] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:12.452160 20941 raft_consensus.cc:2804] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:12.452299 20586 leader_election.cc:304] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:12.452620 20941 raft_consensus.cc:697] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:12.453110 20932 raft_consensus.cc:2804] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:12.453572 20932 raft_consensus.cc:697] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:12.453540 20941 consensus_queue.cc:237] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.454609 20932 consensus_queue.cc:237] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.458691 20771 raft_consensus.cc:2468] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.467106 20520 catalog_manager.cc:5654] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 2 to 3, leader changed from bba8a32c18e84d989607b148c701718b (127.19.131.67) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:12.484961 20520 catalog_manager.cc:5654] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 2 to 3, leader changed from 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:12.744145 20937 raft_consensus.cc:993] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.744565 20939 raft_consensus.cc:1081] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.746026 20621 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52206
I20251214 02:58:12.746616 20621 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:12.746973 20621 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.750792 20937 raft_consensus.cc:993] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: : Instructing follower ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.751190 20939 raft_consensus.cc:1081] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Signalling peer ca992896c00449d49ca22cdf754dbd40 to start an election
I20251214 02:58:12.752534 20622 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
from {username='slave'} at 127.0.0.1:52206
I20251214 02:58:12.753001 20622 raft_consensus.cc:493] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:12.753271 20622 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:12.753926 20621 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.755959 20621 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.756770 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:12.757184 20698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:12.757354 20771 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.757702 20698 raft_consensus.cc:3055] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:12.758021 20698 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:58:12.758733 20698 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.759871 20698 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:12.762219 20771 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.763252 20586 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:12.764055 20941 raft_consensus.cc:2804] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:12.764045 20622 raft_consensus.cc:515] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.764845 20941 raft_consensus.cc:697] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:12.765715 20622 leader_election.cc:290] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Requested vote from peers 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:12.765738 20941 consensus_queue.cc:237] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.767076 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:12.767635 20696 raft_consensus.cc:3055] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:12.767925 20696 raft_consensus.cc:740] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:58:12.768687 20696 consensus_queue.cc:260] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.769481 20698 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 3.
I20251214 02:58:12.769783 20696 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:12.767076 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d813f041d92b4a85a26ddd19f003e84f" candidate_uuid: "ca992896c00449d49ca22cdf754dbd40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:12.774864 20771 raft_consensus.cc:3060] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:12.775413 20520 catalog_manager.cc:5654] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 2 to 3, leader changed from 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:12.778635 20696 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:58:12.779727 20586 leader_election.cc:304] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:12.780848 20932 raft_consensus.cc:2804] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:12.781296 20932 raft_consensus.cc:697] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Leader. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:12.782613 20771 raft_consensus.cc:2468] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992896c00449d49ca22cdf754dbd40 in term 2.
I20251214 02:58:12.782677 20932 consensus_queue.cc:237] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:12.791653 20520 catalog_manager.cc:5654] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 reported cstate change: term changed from 1 to 2, leader changed from 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66) to ca992896c00449d49ca22cdf754dbd40 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "ca992896c00449d49ca22cdf754dbd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:12.959877 20696 raft_consensus.cc:1275] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:12.961050 20932 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:12.975728 20772 raft_consensus.cc:1275] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:12.978296 20932 consensus_queue.cc:1048] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:13.033600 20696 raft_consensus.cc:1275] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:13.034868 20932 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:13.046283 20772 raft_consensus.cc:1275] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:13.047514 20956 consensus_queue.cc:1048] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:13.174928 20696 raft_consensus.cc:1275] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:13.176126 20956 consensus_queue.cc:1048] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:13.185130 20772 raft_consensus.cc:1275] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:13.186473 20934 consensus_queue.cc:1048] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:13.229341 20696 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:13.230670 20941 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:13.241619 20772 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Refusing update from remote peer ca992896c00449d49ca22cdf754dbd40: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:13.243151 20932 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:14.229413 20935 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:14.229959 20935 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:14.232718 20935 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:14.233512 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40" is_pre_election: true
I20251214 02:58:14.233745 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75" is_pre_election: true
I20251214 02:58:14.234163 20622 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 4.
I20251214 02:58:14.234339 20696 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 4.
I20251214 02:58:14.235289 20737 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:14.235996 20935 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:58:14.236343 20935 raft_consensus.cc:493] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:14.236601 20935 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:58:14.243098 20935 raft_consensus.cc:515] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:14.244647 20935 leader_election.cc:290] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 5 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:14.245345 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:14.245676 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e982dce2ff7648758cebe75226ebf7b5" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:14.245908 20622 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:58:14.246212 20696 raft_consensus.cc:3060] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:58:14.252243 20622 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 5.
I20251214 02:58:14.253335 20696 raft_consensus.cc:2468] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 5.
I20251214 02:58:14.253285 20737 leader_election.cc:304] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bba8a32c18e84d989607b148c701718b, ca992896c00449d49ca22cdf754dbd40; no voters:
I20251214 02:58:14.254225 20935 raft_consensus.cc:2804] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:58:14.255224 20935 raft_consensus.cc:697] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 5 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:14.255985 20935 consensus_queue.cc:237] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:14.264252 20520 catalog_manager.cc:5654] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:14.659166 20696 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 5 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:58:14.660529 20935 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:14.674194 20622 raft_consensus.cc:1275] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 5 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:58:14.676949 20963 consensus_queue.cc:1048] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:15.445036 19981 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20251214 02:58:15.445621 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:15.452226 20622 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:52224
I20251214 02:58:15.452667 20622 raft_consensus.cc:606] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Received request to transfer leadership to TS 6b95b11145e4473e94b1313fd2eabf75
I20251214 02:58:15.454777 20622 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:52224
I20251214 02:58:15.455152 20622 raft_consensus.cc:606] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Received request to transfer leadership to TS 6b95b11145e4473e94b1313fd2eabf75
I20251214 02:58:15.457018 20622 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a"
dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
mode: GRACEFUL
new_leader_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:52224
I20251214 02:58:15.457391 20622 raft_consensus.cc:606] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Received request to transfer leadership to TS bba8a32c18e84d989607b148c701718b
I20251214 02:58:15.458313 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251214 02:58:15.458877 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:15.613310 20968 raft_consensus.cc:993] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: : Instructing follower bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:15.613729 20976 raft_consensus.cc:1081] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Signalling peer bba8a32c18e84d989607b148c701718b to start an election
I20251214 02:58:15.615039 20772 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a"
dest_uuid: "bba8a32c18e84d989607b148c701718b"
from {username='slave'} at 127.0.0.1:55796
I20251214 02:58:15.615474 20772 raft_consensus.cc:493] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:15.615717 20772 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.620218 20772 raft_consensus.cc:515] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.621518 20772 leader_election.cc:290] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435)
I20251214 02:58:15.622402 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:15.622499 20696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b61e7d0235f247069c74b83373756b4a" candidate_uuid: "bba8a32c18e84d989607b148c701718b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
I20251214 02:58:15.623039 20622 raft_consensus.cc:3055] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:15.623076 20696 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.623456 20622 raft_consensus.cc:740] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:15.624128 20622 consensus_queue.cc:260] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.625273 20622 raft_consensus.cc:3060] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.629324 20696 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 3.
I20251214 02:58:15.630621 20735 leader_election.cc:304] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:58:15.631103 20622 raft_consensus.cc:2468] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bba8a32c18e84d989607b148c701718b in term 3.
I20251214 02:58:15.631356 20972 raft_consensus.cc:2804] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:15.631827 20972 raft_consensus.cc:697] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Becoming Leader. State: Replica: bba8a32c18e84d989607b148c701718b, State: Running, Role: LEADER
I20251214 02:58:15.632692 20972 consensus_queue.cc:237] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.639955 20519 catalog_manager.cc:5654] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b reported cstate change: term changed from 2 to 3, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to bba8a32c18e84d989607b148c701718b (127.19.131.67). New cstate: current_term: 3 leader_uuid: "bba8a32c18e84d989607b148c701718b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:15.731508 20976 raft_consensus.cc:993] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: : Instructing follower 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:58:15.731964 20968 raft_consensus.cc:1081] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Signalling peer 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:58:15.732985 20968 raft_consensus.cc:993] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: : Instructing follower 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:58:15.733299 20696 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:59772
I20251214 02:58:15.733387 20976 raft_consensus.cc:1081] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Signalling peer 6b95b11145e4473e94b1313fd2eabf75 to start an election
I20251214 02:58:15.733752 20696 raft_consensus.cc:493] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:15.734051 20696 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.734786 20698 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1"
dest_uuid: "6b95b11145e4473e94b1313fd2eabf75"
from {username='slave'} at 127.0.0.1:59772
I20251214 02:58:15.735163 20698 raft_consensus.cc:493] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:58:15.735410 20698 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:15.739154 20696 raft_consensus.cc:515] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:58:15.740317 20698 raft_consensus.cc:515] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.741180 20696 leader_election.cc:290] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 3 election: Requested vote from peers bba8a32c18e84d989607b148c701718b (127.19.131.67:46875), ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613)
I20251214 02:58:15.741659 20698 leader_election.cc:290] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 4 election: Requested vote from peers ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613), bba8a32c18e84d989607b148c701718b (127.19.131.67:46875)
I20251214 02:58:15.741816 20772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:15.742333 20772 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.743016 20771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bba8a32c18e84d989607b148c701718b"
I20251214 02:58:15.743113 20622 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a942f14898c4cfaabc38f4ada704ead" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:15.743531 20771 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:15.743633 20622 raft_consensus.cc:3055] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:15.743762 20621 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0900b8afa00f4d819740e61db7e435f1" candidate_uuid: "6b95b11145e4473e94b1313fd2eabf75" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ca992896c00449d49ca22cdf754dbd40"
I20251214 02:58:15.744060 20622 raft_consensus.cc:740] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:15.744269 20621 raft_consensus.cc:3055] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:58:15.744573 20621 raft_consensus.cc:740] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ca992896c00449d49ca22cdf754dbd40, State: Running, Role: LEADER
I20251214 02:58:15.744808 20622 consensus_queue.cc:260] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:58:15.745267 20621 consensus_queue.cc:260] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.745908 20622 raft_consensus.cc:3060] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:15.746434 20621 raft_consensus.cc:3060] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:58:15.748703 20772 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 3.
I20251214 02:58:15.749953 20661 leader_election.cc:304] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:58:15.750726 20771 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 4.
I20251214 02:58:15.750764 20939 raft_consensus.cc:2804] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:15.751861 20661 leader_election.cc:304] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b95b11145e4473e94b1313fd2eabf75, bba8a32c18e84d989607b148c701718b; no voters:
I20251214 02:58:15.752851 20621 raft_consensus.cc:2468] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 4.
I20251214 02:58:15.753072 20622 raft_consensus.cc:2468] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b95b11145e4473e94b1313fd2eabf75 in term 3.
I20251214 02:58:15.755098 20984 raft_consensus.cc:2804] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:58:15.755473 20984 raft_consensus.cc:697] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:58:15.755689 20939 raft_consensus.cc:697] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 3 LEADER]: Becoming Leader. State: Replica: 6b95b11145e4473e94b1313fd2eabf75, State: Running, Role: LEADER
I20251214 02:58:15.756115 20984 consensus_queue.cc:237] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } }
I20251214 02:58:15.756472 20939 consensus_queue.cc:237] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } }
I20251214 02:58:15.763502 20519 catalog_manager.cc:5654] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 2 to 3, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:15.773677 20519 catalog_manager.cc:5654] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 reported cstate change: term changed from 3 to 4, leader changed from ca992896c00449d49ca22cdf754dbd40 (127.19.131.65) to 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "6b95b11145e4473e94b1313fd2eabf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:16.060567 20621 raft_consensus.cc:1275] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:16.061702 20972 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:16.072340 20698 raft_consensus.cc:1275] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Refusing update from remote peer bba8a32c18e84d989607b148c701718b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:16.074595 20935 consensus_queue.cc:1048] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95b11145e4473e94b1313fd2eabf75" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:16.147662 20771 raft_consensus.cc:1275] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:16.148728 20939 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:16.156661 20621 raft_consensus.cc:1275] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:16.158085 20983 consensus_queue.cc:1048] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:16.289836 20622 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:58:16.290912 20983 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca992896c00449d49ca22cdf754dbd40" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:16.298599 20771 raft_consensus.cc:1275] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Refusing update from remote peer 6b95b11145e4473e94b1313fd2eabf75: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:58:16.299985 20939 consensus_queue.cc:1048] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba8a32c18e84d989607b148c701718b" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:16.459801 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:16.466583 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:16.467056 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:17.472031 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:17.480065 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:17.480495 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:18.481868 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:18.487454 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:18.487810 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:19.488622 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:19.493799 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:19.494210 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:20.495059 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:20.500725 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:20.501169 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:21.502017 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:21.507541 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:21.507944 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:22.508769 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:22.513873 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:22.514287 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:23.515177 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:23.520490 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:23.520859 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:24.521625 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:58:24.527066 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251214 02:58:24.527436 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:58:25.553063 19981 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20251214 02:58:25.553507 19981 tablet_server.cc:178] TabletServer@127.19.131.65:0 shutting down...
I20251214 02:58:25.581332 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:58:25.581966 19981 tablet_replica.cc:333] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.582728 19981 raft_consensus.cc:2243] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.583283 19981 raft_consensus.cc:2272] T e08c401f218e44d6975b50cb229454eb P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.585160 19981 tablet_replica.cc:333] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.585644 19981 raft_consensus.cc:2243] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.586367 19981 raft_consensus.cc:2272] T d50956b58a724d2b86316a213bfc6314 P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.588043 19981 tablet_replica.cc:333] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.588536 19981 raft_consensus.cc:2243] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.589265 19981 raft_consensus.cc:2272] T 2d2bdcdbe4244334b95ed4eabf856a39 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.590898 19981 tablet_replica.cc:333] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.591332 19981 raft_consensus.cc:2243] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 2 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.592339 19981 raft_consensus.cc:2272] T d813f041d92b4a85a26ddd19f003e84f P ca992896c00449d49ca22cdf754dbd40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.594442 19981 tablet_replica.cc:333] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.594950 19981 raft_consensus.cc:2243] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.595366 19981 raft_consensus.cc:2272] T 1a942f14898c4cfaabc38f4ada704ead P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.597009 19981 tablet_replica.cc:333] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.597438 19981 raft_consensus.cc:2243] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.597815 19981 raft_consensus.cc:2272] T 948bafb5a8844d2cabfe1d2eb6c5d785 P ca992896c00449d49ca22cdf754dbd40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.599500 19981 tablet_replica.cc:333] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.599951 19981 raft_consensus.cc:2243] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.600334 19981 raft_consensus.cc:2272] T e982dce2ff7648758cebe75226ebf7b5 P ca992896c00449d49ca22cdf754dbd40 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.601929 19981 tablet_replica.cc:333] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.602376 19981 raft_consensus.cc:2243] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.602782 19981 raft_consensus.cc:2272] T 0900b8afa00f4d819740e61db7e435f1 P ca992896c00449d49ca22cdf754dbd40 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.604302 19981 tablet_replica.cc:333] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.604737 19981 raft_consensus.cc:2243] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.605567 19981 raft_consensus.cc:2272] T a29b17e78cf24d5987135f72e68b6822 P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.607653 19981 tablet_replica.cc:333] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40: stopping tablet replica
I20251214 02:58:25.608199 19981 raft_consensus.cc:2243] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.608778 19981 raft_consensus.cc:2272] T b61e7d0235f247069c74b83373756b4a P ca992896c00449d49ca22cdf754dbd40 [term 3 FOLLOWER]: Raft consensus is shut down!
W20251214 02:58:25.609396 20737 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251214 02:58:25.612457 20737 consensus_peers.cc:597] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b -> Peer ca992896c00449d49ca22cdf754dbd40 (127.19.131.65:36613): Couldn't send request to peer ca992896c00449d49ca22cdf754dbd40. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:25.638154 19981 tablet_server.cc:195] TabletServer@127.19.131.65:0 shutdown complete.
I20251214 02:58:25.657083 19981 tablet_server.cc:178] TabletServer@127.19.131.66:0 shutting down...
W20251214 02:58:25.674147 20735 consensus_peers.cc:597] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b -> Peer 6b95b11145e4473e94b1313fd2eabf75 (127.19.131.66:46435): Couldn't send request to peer 6b95b11145e4473e94b1313fd2eabf75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:25.680148 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:58:25.680830 19981 tablet_replica.cc:333] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.681650 19981 raft_consensus.cc:2243] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.682313 19981 raft_consensus.cc:2272] T e08c401f218e44d6975b50cb229454eb P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.684190 19981 tablet_replica.cc:333] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.684650 19981 raft_consensus.cc:2243] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.685350 19981 raft_consensus.cc:2272] T 0900b8afa00f4d819740e61db7e435f1 P 6b95b11145e4473e94b1313fd2eabf75 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.686954 19981 tablet_replica.cc:333] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.687419 19981 raft_consensus.cc:2243] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.687805 19981 raft_consensus.cc:2272] T 2d2bdcdbe4244334b95ed4eabf856a39 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.689343 19981 tablet_replica.cc:333] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.689760 19981 raft_consensus.cc:2243] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.690091 19981 raft_consensus.cc:2272] T a29b17e78cf24d5987135f72e68b6822 P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.691666 19981 tablet_replica.cc:333] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.692082 19981 raft_consensus.cc:2243] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.692414 19981 raft_consensus.cc:2272] T b61e7d0235f247069c74b83373756b4a P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.693933 19981 tablet_replica.cc:333] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.694362 19981 raft_consensus.cc:2243] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.694751 19981 raft_consensus.cc:2272] T d813f041d92b4a85a26ddd19f003e84f P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.696349 19981 tablet_replica.cc:333] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.696779 19981 raft_consensus.cc:2243] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.697463 19981 raft_consensus.cc:2272] T 1a942f14898c4cfaabc38f4ada704ead P 6b95b11145e4473e94b1313fd2eabf75 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.699069 19981 tablet_replica.cc:333] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.699476 19981 raft_consensus.cc:2243] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.700129 19981 raft_consensus.cc:2272] T 948bafb5a8844d2cabfe1d2eb6c5d785 P 6b95b11145e4473e94b1313fd2eabf75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.701809 19981 tablet_replica.cc:333] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.702221 19981 raft_consensus.cc:2243] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.702636 19981 raft_consensus.cc:2272] T e982dce2ff7648758cebe75226ebf7b5 P 6b95b11145e4473e94b1313fd2eabf75 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.704166 19981 tablet_replica.cc:333] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75: stopping tablet replica
I20251214 02:58:25.704540 19981 raft_consensus.cc:2243] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.704890 19981 raft_consensus.cc:2272] T d50956b58a724d2b86316a213bfc6314 P 6b95b11145e4473e94b1313fd2eabf75 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.731549 19981 tablet_server.cc:195] TabletServer@127.19.131.66:0 shutdown complete.
I20251214 02:58:25.749491 19981 tablet_server.cc:178] TabletServer@127.19.131.67:0 shutting down...
I20251214 02:58:25.773502 19981 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251214 02:58:25.774127 19981 tablet_replica.cc:333] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.774703 19981 raft_consensus.cc:2243] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 2 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.775450 19981 raft_consensus.cc:2272] T e08c401f218e44d6975b50cb229454eb P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.777263 19981 tablet_replica.cc:333] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.777724 19981 raft_consensus.cc:2243] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.778085 19981 raft_consensus.cc:2272] T 2d2bdcdbe4244334b95ed4eabf856a39 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.779716 19981 tablet_replica.cc:333] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.780156 19981 raft_consensus.cc:2243] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.780507 19981 raft_consensus.cc:2272] T d50956b58a724d2b86316a213bfc6314 P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.782073 19981 tablet_replica.cc:333] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.782514 19981 raft_consensus.cc:2243] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.782953 19981 raft_consensus.cc:2272] T 0900b8afa00f4d819740e61db7e435f1 P bba8a32c18e84d989607b148c701718b [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.784380 19981 tablet_replica.cc:333] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.784811 19981 raft_consensus.cc:2243] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.785135 19981 raft_consensus.cc:2272] T 948bafb5a8844d2cabfe1d2eb6c5d785 P bba8a32c18e84d989607b148c701718b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.786968 19981 tablet_replica.cc:333] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.787408 19981 raft_consensus.cc:2243] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 5 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.788102 19981 raft_consensus.cc:2272] T e982dce2ff7648758cebe75226ebf7b5 P bba8a32c18e84d989607b148c701718b [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.789588 19981 tablet_replica.cc:333] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.790236 19981 raft_consensus.cc:2243] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 3 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.790961 19981 raft_consensus.cc:2272] T b61e7d0235f247069c74b83373756b4a P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.792387 19981 tablet_replica.cc:333] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.792770 19981 raft_consensus.cc:2243] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.793087 19981 raft_consensus.cc:2272] T a29b17e78cf24d5987135f72e68b6822 P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.794529 19981 tablet_replica.cc:333] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.794929 19981 raft_consensus.cc:2243] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.795271 19981 raft_consensus.cc:2272] T d813f041d92b4a85a26ddd19f003e84f P bba8a32c18e84d989607b148c701718b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.796703 19981 tablet_replica.cc:333] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b: stopping tablet replica
I20251214 02:58:25.797061 19981 raft_consensus.cc:2243] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:25.797437 19981 raft_consensus.cc:2272] T 1a942f14898c4cfaabc38f4ada704ead P bba8a32c18e84d989607b148c701718b [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.824216 19981 tablet_server.cc:195] TabletServer@127.19.131.67:0 shutdown complete.
I20251214 02:58:25.841997 19981 master.cc:561] Master@127.19.131.126:33021 shutting down...
I20251214 02:58:25.861244 19981 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 1 LEADER]: Raft consensus shutting down.
I20251214 02:58:25.861768 19981 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:25.862043 19981 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5937feb5e09e47f7a7ce495e020158cd: stopping tablet replica
I20251214 02:58:25.897734 19981 master.cc:583] Master@127.19.131.126:33021 shutdown complete.
6b95b11145e4473e94b1313fd2eabf75 4
bba8a32c18e84d989607b148c701718b 1
ca992896c00449d49ca22cdf754dbd40 5
6b95b11145e4473e94b1313fd2eabf75 4
bba8a32c18e84d989607b148c701718b 3
ca992896c00449d49ca22cdf754dbd40 3
6b95b11145e4473e94b1313fd2eabf75 1
bba8a32c18e84d989607b148c701718b 2
ca992896c00449d49ca22cdf754dbd40 7
6b95b11145e4473e94b1313fd2eabf75 3
bba8a32c18e84d989607b148c701718b 3
ca992896c00449d49ca22cdf754dbd40 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28788 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20251214 02:58:25.938257 19981 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.131.126:41013
I20251214 02:58:25.939396 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:58:25.945037 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:25.945276 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:25.945783 21009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:25.946920 19981 server_base.cc:1047] running on GCE node
I20251214 02:58:25.947757 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:58:25.947937 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:58:25.948046 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681105948036 us; error 0 us; skew 500 ppm
I20251214 02:58:25.948488 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:58:25.950691 19981 webserver.cc:492] Webserver started at http://127.19.131.126:38015/ using document root <none> and password file <none>
I20251214 02:58:25.951125 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:58:25.951278 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:58:25.951489 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:58:25.952533 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/master-0-root/instance:
uuid: "86f60dbd6dbf443abf2563a85669a2b8"
format_stamp: "Formatted at 2025-12-14 02:58:25 on dist-test-slave-8pjb"
I20251214 02:58:25.956698 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251214 02:58:25.959771 21014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:25.960420 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251214 02:58:25.960659 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/master-0-root
uuid: "86f60dbd6dbf443abf2563a85669a2b8"
format_stamp: "Formatted at 2025-12-14 02:58:25 on dist-test-slave-8pjb"
I20251214 02:58:25.960901 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:58:26.002691 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:58:26.003875 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:58:26.039933 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.126:41013
I20251214 02:58:26.040023 21065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.126:41013 every 8 connection(s)
I20251214 02:58:26.043639 21066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.053345 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Bootstrap starting.
I20251214 02:58:26.057839 21066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:26.061762 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: No bootstrap required, opened a new log
I20251214 02:58:26.063881 21066 raft_consensus.cc:359] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER }
I20251214 02:58:26.064299 21066 raft_consensus.cc:385] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:26.064486 21066 raft_consensus.cc:740] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f60dbd6dbf443abf2563a85669a2b8, State: Initialized, Role: FOLLOWER
I20251214 02:58:26.064946 21066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER }
I20251214 02:58:26.065356 21066 raft_consensus.cc:399] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251214 02:58:26.065541 21066 raft_consensus.cc:493] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251214 02:58:26.065757 21066 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:26.069649 21066 raft_consensus.cc:515] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER }
I20251214 02:58:26.070087 21066 leader_election.cc:304] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f60dbd6dbf443abf2563a85669a2b8; no voters:
I20251214 02:58:26.071120 21066 leader_election.cc:290] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251214 02:58:26.071475 21069 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:26.072721 21069 raft_consensus.cc:697] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 86f60dbd6dbf443abf2563a85669a2b8, State: Running, Role: LEADER
I20251214 02:58:26.073316 21069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER }
I20251214 02:58:26.074003 21066 sys_catalog.cc:565] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 02:58:26.077862 21070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f60dbd6dbf443abf2563a85669a2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER } }
I20251214 02:58:26.079413 21070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [sys.catalog]: This master's current role is: LEADER
I20251214 02:58:26.078852 21071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f60dbd6dbf443abf2563a85669a2b8. Latest consensus state: current_term: 1 leader_uuid: "86f60dbd6dbf443abf2563a85669a2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f60dbd6dbf443abf2563a85669a2b8" member_type: VOTER } }
I20251214 02:58:26.080497 21071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8 [sys.catalog]: This master's current role is: LEADER
I20251214 02:58:26.084806 21078 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 02:58:26.090065 21078 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 02:58:26.091094 19981 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251214 02:58:26.099915 21078 catalog_manager.cc:1357] Generated new cluster ID: 6cdeb4b8127244649000735bcdad38f1
I20251214 02:58:26.100205 21078 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 02:58:26.119045 21078 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 02:58:26.120267 21078 catalog_manager.cc:1514] Loading token signing keys...
I20251214 02:58:26.131768 21078 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Generated new TSK 0
I20251214 02:58:26.132308 21078 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 02:58:26.157608 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:58:26.164402 21088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:26.164861 21087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:26.166323 19981 server_base.cc:1047] running on GCE node
W20251214 02:58:26.166364 21090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:26.167418 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:58:26.167603 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:58:26.167752 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681106167738 us; error 0 us; skew 500 ppm
I20251214 02:58:26.168285 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:58:26.170749 19981 webserver.cc:492] Webserver started at http://127.19.131.65:42213/ using document root <none> and password file <none>
I20251214 02:58:26.171202 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:58:26.171367 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:58:26.171612 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:58:26.172631 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/instance:
uuid: "721d604130924d939d291fe3eef8f6d1"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.176834 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20251214 02:58:26.180011 21095 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.180696 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 02:58:26.180967 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root
uuid: "721d604130924d939d291fe3eef8f6d1"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.181246 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:58:26.199095 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:58:26.200296 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:58:26.201728 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:58:26.203913 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:58:26.204106 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.204319 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:58:26.204461 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.243072 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.65:38855
I20251214 02:58:26.243165 21157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.65:38855 every 8 connection(s)
I20251214 02:58:26.248618 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:58:26.254215 21162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:26.256680 21163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:26.258394 21165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:26.259351 19981 server_base.cc:1047] running on GCE node
I20251214 02:58:26.260253 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:58:26.260454 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:58:26.260617 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681106260599 us; error 0 us; skew 500 ppm
I20251214 02:58:26.261106 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:58:26.262420 21158 heartbeater.cc:344] Connected to a master server at 127.19.131.126:41013
I20251214 02:58:26.262812 21158 heartbeater.cc:461] Registering TS with master...
I20251214 02:58:26.263695 21158 heartbeater.cc:507] Master 127.19.131.126:41013 requested a full tablet report, sending...
I20251214 02:58:26.263857 19981 webserver.cc:492] Webserver started at http://127.19.131.66:38737/ using document root <none> and password file <none>
I20251214 02:58:26.264477 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:58:26.264663 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:58:26.264961 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:58:26.265691 21031 ts_manager.cc:194] Registered new tserver with Master: 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:26.266116 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/instance:
uuid: "668ad4f30bd649c88a034bbab9f13b20"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.267401 21031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51636
I20251214 02:58:26.270875 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251214 02:58:26.274012 21170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.274667 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251214 02:58:26.274916 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root
uuid: "668ad4f30bd649c88a034bbab9f13b20"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.275156 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:58:26.304801 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:58:26.305976 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:58:26.307411 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:58:26.309538 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:58:26.309705 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.309931 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:58:26.310068 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.348129 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.66:44049
I20251214 02:58:26.348258 21232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.66:44049 every 8 connection(s)
I20251214 02:58:26.352556 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 02:58:26.359047 21236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:26.359839 21237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:26.362308 21239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:26.362640 19981 server_base.cc:1047] running on GCE node
I20251214 02:58:26.363618 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:58:26.363819 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:58:26.363849 21233 heartbeater.cc:344] Connected to a master server at 127.19.131.126:41013
I20251214 02:58:26.364020 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681106364003 us; error 0 us; skew 500 ppm
I20251214 02:58:26.364267 21233 heartbeater.cc:461] Registering TS with master...
I20251214 02:58:26.364594 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:58:26.364909 21233 heartbeater.cc:507] Master 127.19.131.126:41013 requested a full tablet report, sending...
I20251214 02:58:26.366737 21031 ts_manager.cc:194] Registered new tserver with Master: 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:26.367146 19981 webserver.cc:492] Webserver started at http://127.19.131.67:35431/ using document root <none> and password file <none>
I20251214 02:58:26.367638 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:58:26.367821 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:58:26.368058 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:58:26.368191 21031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51648
I20251214 02:58:26.369138 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/instance:
uuid: "6e36caefa37d47a5968cffd9e270044f"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.373337 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251214 02:58:26.376523 21244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.377274 19981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251214 02:58:26.377513 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root
uuid: "6e36caefa37d47a5968cffd9e270044f"
format_stamp: "Formatted at 2025-12-14 02:58:26 on dist-test-slave-8pjb"
I20251214 02:58:26.377744 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:58:26.394878 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:58:26.396039 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:58:26.397423 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:58:26.399557 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:58:26.399755 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.399991 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:58:26.400122 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:26.438459 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.67:34525
I20251214 02:58:26.438553 21306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.67:34525 every 8 connection(s)
I20251214 02:58:26.451557 21307 heartbeater.cc:344] Connected to a master server at 127.19.131.126:41013
I20251214 02:58:26.451884 21307 heartbeater.cc:461] Registering TS with master...
I20251214 02:58:26.452597 21307 heartbeater.cc:507] Master 127.19.131.126:41013 requested a full tablet report, sending...
I20251214 02:58:26.454312 21031 ts_manager.cc:194] Registered new tserver with Master: 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:26.454766 19981 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013346516s
I20251214 02:58:26.455976 21031 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51662
I20251214 02:58:26.457486 19981 test_util.cc:276] Using random seed: -1423764620
I20251214 02:58:26.478794 21031 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51668:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 02:58:26.484908 21031 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 02:58:26.676592 21272 tablet_service.cc:1505] Processing CreateTablet for tablet 2734ae274c16411d93b2c95d7fb3e00c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251214 02:58:26.677016 21271 tablet_service.cc:1505] Processing CreateTablet for tablet b832eb0688dc4d6c990d35c712ef6cc7 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251214 02:58:26.678097 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2734ae274c16411d93b2c95d7fb3e00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.679420 21270 tablet_service.cc:1505] Processing CreateTablet for tablet d55cde2542d544c5b99e6113b95910b8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251214 02:58:26.680727 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55cde2542d544c5b99e6113b95910b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.684888 21269 tablet_service.cc:1505] Processing CreateTablet for tablet fb8de75de22740c889d4e6fe5fae1bc6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251214 02:58:26.686200 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8de75de22740c889d4e6fe5fae1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.686858 21196 tablet_service.cc:1505] Processing CreateTablet for tablet d55cde2542d544c5b99e6113b95910b8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251214 02:58:26.687712 21198 tablet_service.cc:1505] Processing CreateTablet for tablet 2734ae274c16411d93b2c95d7fb3e00c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251214 02:58:26.688215 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55cde2542d544c5b99e6113b95910b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.687861 21194 tablet_service.cc:1505] Processing CreateTablet for tablet e91314b7fb3c4dbd8db2eebcf16d820f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251214 02:58:26.688913 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2734ae274c16411d93b2c95d7fb3e00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.690176 21195 tablet_service.cc:1505] Processing CreateTablet for tablet fb8de75de22740c889d4e6fe5fae1bc6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251214 02:58:26.690267 21268 tablet_service.cc:1505] Processing CreateTablet for tablet e91314b7fb3c4dbd8db2eebcf16d820f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251214 02:58:26.691538 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8de75de22740c889d4e6fe5fae1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.691658 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91314b7fb3c4dbd8db2eebcf16d820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.695758 21197 tablet_service.cc:1505] Processing CreateTablet for tablet b832eb0688dc4d6c990d35c712ef6cc7 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251214 02:58:26.697082 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b832eb0688dc4d6c990d35c712ef6cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.698968 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91314b7fb3c4dbd8db2eebcf16d820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.699828 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b832eb0688dc4d6c990d35c712ef6cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.705214 21322 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:26.725297 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 2953f784a3f34b61a952ba296007ae31 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251214 02:58:26.755993 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2953f784a3f34b61a952ba296007ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.737269 21267 tablet_service.cc:1505] Processing CreateTablet for tablet 51f125bb79074d9fbb6c280fa03dc07e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251214 02:58:26.757499 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f125bb79074d9fbb6c280fa03dc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.757462 21191 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddd57244a03440195bc9142e5d4344a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251214 02:58:26.758663 21322 tablet_bootstrap.cc:654] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:26.758853 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddd57244a03440195bc9142e5d4344a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.745323 21271 tablet_service.cc:1505] Processing CreateTablet for tablet c8ab1ed25694443cb4efebfc35e5205e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251214 02:58:26.760994 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ab1ed25694443cb4efebfc35e5205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.742337 21193 tablet_service.cc:1505] Processing CreateTablet for tablet c8ab1ed25694443cb4efebfc35e5205e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251214 02:58:26.753175 21266 tablet_service.cc:1505] Processing CreateTablet for tablet b8b9d49c54144b37a99555815a60e1e4 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251214 02:58:26.749991 21192 tablet_service.cc:1505] Processing CreateTablet for tablet b8b9d49c54144b37a99555815a60e1e4 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251214 02:58:26.765620 21266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b9d49c54144b37a99555815a60e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.736176 21195 tablet_service.cc:1505] Processing CreateTablet for tablet 51f125bb79074d9fbb6c280fa03dc07e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251214 02:58:26.766510 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ab1ed25694443cb4efebfc35e5205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.767149 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f125bb79074d9fbb6c280fa03dc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.768754 21192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b9d49c54144b37a99555815a60e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.719264 21272 tablet_service.cc:1505] Processing CreateTablet for tablet 2953f784a3f34b61a952ba296007ae31 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251214 02:58:26.782585 21123 tablet_service.cc:1505] Processing CreateTablet for tablet 2734ae274c16411d93b2c95d7fb3e00c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251214 02:58:26.783957 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2734ae274c16411d93b2c95d7fb3e00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.783638 21198 tablet_service.cc:1505] Processing CreateTablet for tablet 639e38cdc0a74102bbcd86211abe87a0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251214 02:58:26.784870 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639e38cdc0a74102bbcd86211abe87a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.787312 21122 tablet_service.cc:1505] Processing CreateTablet for tablet b832eb0688dc4d6c990d35c712ef6cc7 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251214 02:58:26.788553 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b832eb0688dc4d6c990d35c712ef6cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.791687 21121 tablet_service.cc:1505] Processing CreateTablet for tablet d55cde2542d544c5b99e6113b95910b8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251214 02:58:26.792987 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55cde2542d544c5b99e6113b95910b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.795703 21120 tablet_service.cc:1505] Processing CreateTablet for tablet fb8de75de22740c889d4e6fe5fae1bc6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251214 02:58:26.797046 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8de75de22740c889d4e6fe5fae1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.796818 21190 tablet_service.cc:1505] Processing CreateTablet for tablet a3c57218e1ae40dfbdcec8dd228432ca (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251214 02:58:26.798125 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c57218e1ae40dfbdcec8dd228432ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.799608 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2953f784a3f34b61a952ba296007ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.801337 21115 tablet_service.cc:1505] Processing CreateTablet for tablet b8b9d49c54144b37a99555815a60e1e4 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251214 02:58:26.801733 21265 tablet_service.cc:1505] Processing CreateTablet for tablet a3c57218e1ae40dfbdcec8dd228432ca (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251214 02:58:26.802667 21115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b9d49c54144b37a99555815a60e1e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.803064 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c57218e1ae40dfbdcec8dd228432ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.800757 21269 tablet_service.cc:1505] Processing CreateTablet for tablet 639e38cdc0a74102bbcd86211abe87a0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251214 02:58:26.803668 21116 tablet_service.cc:1505] Processing CreateTablet for tablet c8ab1ed25694443cb4efebfc35e5205e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251214 02:58:26.804780 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639e38cdc0a74102bbcd86211abe87a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.804926 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ab1ed25694443cb4efebfc35e5205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.800278 21119 tablet_service.cc:1505] Processing CreateTablet for tablet e91314b7fb3c4dbd8db2eebcf16d820f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251214 02:58:26.805951 21119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91314b7fb3c4dbd8db2eebcf16d820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.806921 21117 tablet_service.cc:1505] Processing CreateTablet for tablet 51f125bb79074d9fbb6c280fa03dc07e (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251214 02:58:26.807495 21268 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddd57244a03440195bc9142e5d4344a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251214 02:58:26.808184 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f125bb79074d9fbb6c280fa03dc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.808677 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddd57244a03440195bc9142e5d4344a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.811367 21323 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:26.811686 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 2953f784a3f34b61a952ba296007ae31 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251214 02:58:26.812896 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2953f784a3f34b61a952ba296007ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.816694 21114 tablet_service.cc:1505] Processing CreateTablet for tablet 0ddd57244a03440195bc9142e5d4344a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251214 02:58:26.818713 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddd57244a03440195bc9142e5d4344a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.846215 21323 tablet_bootstrap.cc:654] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:26.861119 21270 tablet_service.cc:1505] Processing CreateTablet for tablet 57d922c88eca4233b3dd4d314fac9e4a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251214 02:58:26.862452 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d922c88eca4233b3dd4d314fac9e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.911680 21267 tablet_service.cc:1505] Processing CreateTablet for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251214 02:58:26.912997 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.915524 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251214 02:58:26.916800 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.921268 21195 tablet_service.cc:1505] Processing CreateTablet for tablet f692dd0fa4b64450b9808a14da2b1c29 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251214 02:58:26.922510 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f692dd0fa4b64450b9808a14da2b1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.947108 21196 tablet_service.cc:1505] Processing CreateTablet for tablet 57d922c88eca4233b3dd4d314fac9e4a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251214 02:58:26.948472 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d922c88eca4233b3dd4d314fac9e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:26.980564 21116 tablet_service.cc:1505] Processing CreateTablet for tablet 639e38cdc0a74102bbcd86211abe87a0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251214 02:58:26.997993 21120 tablet_service.cc:1505] Processing CreateTablet for tablet a3c57218e1ae40dfbdcec8dd228432ca (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251214 02:58:27.008855 21323 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:27.009464 21323 ts_tablet_manager.cc:1403] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.198s user 0.016s sys 0.040s
I20251214 02:58:27.012248 21323 raft_consensus.cc:359] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.013514 21323 raft_consensus.cc:385] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.013989 21323 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.015275 21323 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.012205 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639e38cdc0a74102bbcd86211abe87a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.016685 21121 tablet_service.cc:1505] Processing CreateTablet for tablet 57d922c88eca4233b3dd4d314fac9e4a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251214 02:58:27.018046 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d922c88eca4233b3dd4d314fac9e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.025439 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251214 02:58:27.028380 21327 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:27.029255 21322 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:27.029723 21322 ts_tablet_manager.cc:1403] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.325s user 0.024s sys 0.028s
I20251214 02:58:27.032380 21322 raft_consensus.cc:359] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.033068 21322 raft_consensus.cc:385] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.033380 21322 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.034217 21327 tablet_bootstrap.cc:654] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.034397 21322 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.033167 21119 tablet_service.cc:1505] Processing CreateTablet for tablet f692dd0fa4b64450b9808a14da2b1c29 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251214 02:58:27.036796 21307 heartbeater.cc:499] Master 127.19.131.126:41013 was elected leader, sending a full tablet report...
I20251214 02:58:27.037503 21322 ts_tablet_manager.cc:1434] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251214 02:58:27.038480 21322 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:27.044092 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c57218e1ae40dfbdcec8dd228432ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.047020 21233 heartbeater.cc:499] Master 127.19.131.126:41013 was elected leader, sending a full tablet report...
I20251214 02:58:27.047896 21323 ts_tablet_manager.cc:1434] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.038s user 0.005s sys 0.005s
I20251214 02:58:27.048723 21323 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:27.053193 21323 tablet_bootstrap.cc:654] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.053467 21322 tablet_bootstrap.cc:654] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.055322 21327 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:27.055930 21327 ts_tablet_manager.cc:1403] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s
W20251214 02:58:27.088236 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:27.112474 21119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f692dd0fa4b64450b9808a14da2b1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.117565 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.119699 21122 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae31f8b580840eb863e4d400367c1be (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251214 02:58:27.121412 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae31f8b580840eb863e4d400367c1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.129376 21123 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba4da4df7bc466aa9cc42fa304335d8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251214 02:58:27.130784 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba4da4df7bc466aa9cc42fa304335d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.129379 21115 tablet_service.cc:1505] Processing CreateTablet for tablet 275e8242241b4fd386a7c5c738a2c903 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251214 02:58:27.135617 21115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e8242241b4fd386a7c5c738a2c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.139509 21114 tablet_service.cc:1505] Processing CreateTablet for tablet 54e50911547a447cb80620e5c0ebb0b9 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251214 02:58:27.140753 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e50911547a447cb80620e5c0ebb0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.151562 21117 tablet_service.cc:1505] Processing CreateTablet for tablet e0e04af2e6054ecabda3186cef55253f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251214 02:58:27.152935 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e04af2e6054ecabda3186cef55253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.155972 21189 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce7181915c14c369bac4b962d34913a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251214 02:58:27.157572 21189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce7181915c14c369bac4b962d34913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.165280 21121 tablet_service.cc:1505] Processing CreateTablet for tablet 143de581216646f79bece2b9f1259f64 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251214 02:58:27.169122 21190 tablet_service.cc:1505] Processing CreateTablet for tablet 75c047f1d6d94237afc050db2241d659 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251214 02:58:27.170476 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c047f1d6d94237afc050db2241d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.173456 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143de581216646f79bece2b9f1259f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.186193 21191 tablet_service.cc:1505] Processing CreateTablet for tablet 54e50911547a447cb80620e5c0ebb0b9 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251214 02:58:27.187652 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e50911547a447cb80620e5c0ebb0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.194024 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 75c047f1d6d94237afc050db2241d659 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251214 02:58:27.195343 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c047f1d6d94237afc050db2241d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.198273 21192 tablet_service.cc:1505] Processing CreateTablet for tablet e0e04af2e6054ecabda3186cef55253f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251214 02:58:27.199717 21192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e04af2e6054ecabda3186cef55253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.205406 21120 tablet_service.cc:1505] Processing CreateTablet for tablet 96ca056dd7594a23a70918fc54c5eb2a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251214 02:58:27.206692 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ca056dd7594a23a70918fc54c5eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.207757 21193 tablet_service.cc:1505] Processing CreateTablet for tablet 96ca056dd7594a23a70918fc54c5eb2a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251214 02:58:27.209038 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ca056dd7594a23a70918fc54c5eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.215819 21196 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae31f8b580840eb863e4d400367c1be (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251214 02:58:27.217185 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae31f8b580840eb863e4d400367c1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.216995 21195 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba4da4df7bc466aa9cc42fa304335d8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251214 02:58:27.218298 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba4da4df7bc466aa9cc42fa304335d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.215832 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 143de581216646f79bece2b9f1259f64 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251214 02:58:27.218072 21116 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3be94c517348a08049e54abf8fd780 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251214 02:58:27.221838 21198 tablet_service.cc:1505] Processing CreateTablet for tablet 275e8242241b4fd386a7c5c738a2c903 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251214 02:58:27.220036 21197 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3be94c517348a08049e54abf8fd780 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
W20251214 02:58:27.222930 21234 tablet.cc:2378] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 02:58:27.224308 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143de581216646f79bece2b9f1259f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.230906 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e8242241b4fd386a7c5c738a2c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.235071 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3be94c517348a08049e54abf8fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.235882 21263 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3be94c517348a08049e54abf8fd780 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251214 02:58:27.237470 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3be94c517348a08049e54abf8fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.237143 21264 tablet_service.cc:1505] Processing CreateTablet for tablet 96ca056dd7594a23a70918fc54c5eb2a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251214 02:58:27.238480 21264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ca056dd7594a23a70918fc54c5eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.235850 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3be94c517348a08049e54abf8fd780. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.243472 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251214 02:58:27.253602 21120 tablet_service.cc:1505] Processing CreateTablet for tablet 75e354ca56944ff1b7775a8309bca568 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251214 02:58:27.258479 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.261435 21269 tablet_service.cc:1505] Processing CreateTablet for tablet 143de581216646f79bece2b9f1259f64 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251214 02:58:27.262719 21266 tablet_service.cc:1505] Processing CreateTablet for tablet 54e50911547a447cb80620e5c0ebb0b9 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251214 02:58:27.263306 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143de581216646f79bece2b9f1259f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.263612 21272 tablet_service.cc:1505] Processing CreateTablet for tablet 75c047f1d6d94237afc050db2241d659 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251214 02:58:27.261435 21265 tablet_service.cc:1505] Processing CreateTablet for tablet 4ba4da4df7bc466aa9cc42fa304335d8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251214 02:58:27.264623 21267 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae31f8b580840eb863e4d400367c1be (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251214 02:58:27.266288 21268 tablet_service.cc:1505] Processing CreateTablet for tablet 275e8242241b4fd386a7c5c738a2c903 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251214 02:58:27.268128 21270 tablet_service.cc:1505] Processing CreateTablet for tablet f692dd0fa4b64450b9808a14da2b1c29 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251214 02:58:27.270206 21158 heartbeater.cc:499] Master 127.19.131.126:41013 was elected leader, sending a full tablet report...
I20251214 02:58:27.269989 21271 tablet_service.cc:1505] Processing CreateTablet for tablet e0e04af2e6054ecabda3186cef55253f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251214 02:58:27.270754 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e354ca56944ff1b7775a8309bca568. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.272379 21266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e50911547a447cb80620e5c0ebb0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.275586 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e8242241b4fd386a7c5c738a2c903. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.276461 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f692dd0fa4b64450b9808a14da2b1c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.277290 21263 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce7181915c14c369bac4b962d34913a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251214 02:58:27.278090 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae31f8b580840eb863e4d400367c1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.283213 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e04af2e6054ecabda3186cef55253f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.284119 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c047f1d6d94237afc050db2241d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.284837 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba4da4df7bc466aa9cc42fa304335d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.234896 21119 tablet_service.cc:1505] Processing CreateTablet for tablet 2ce7181915c14c369bac4b962d34913a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251214 02:58:27.293730 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce7181915c14c369bac4b962d34913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.295030 21328 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:27.298866 21328 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.299705 21117 tablet_service.cc:1505] Processing CreateTablet for tablet 5751534b61ed4568abd62057238d5fda (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251214 02:58:27.284469 21121 tablet_service.cc:1505] Processing CreateTablet for tablet 9fdf7633f868447a8abd2adfe9b04fc5 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251214 02:58:27.297631 21322 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:27.302174 21322 ts_tablet_manager.cc:1403] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.006s
I20251214 02:58:27.304291 21323 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:27.304832 21323 ts_tablet_manager.cc:1403] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.256s user 0.004s sys 0.005s
I20251214 02:58:27.305269 21322 raft_consensus.cc:359] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.305992 21322 raft_consensus.cc:385] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.307502 21328 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:27.306306 21322 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.312870 21265 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251214 02:58:27.314145 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.294443 21327 raft_consensus.cc:359] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.315379 21327 raft_consensus.cc:385] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.314647 21322 consensus_queue.cc:260] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.315670 21327 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.320060 21323 raft_consensus.cc:359] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.320659 21269 tablet_service.cc:1505] Processing CreateTablet for tablet 75e354ca56944ff1b7775a8309bca568 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251214 02:58:27.321025 21327 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.329382 21323 raft_consensus.cc:385] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.329769 21323 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.330117 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e354ca56944ff1b7775a8309bca568. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.338444 21322 ts_tablet_manager.cc:1434] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20251214 02:58:27.343521 21190 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251214 02:58:27.347096 21265 tablet_service.cc:1505] Processing CreateTablet for tablet 9fdf7633f868447a8abd2adfe9b04fc5 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251214 02:58:27.359371 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdf7633f868447a8abd2adfe9b04fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.355719 21323 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.375000 21271 tablet_service.cc:1505] Processing CreateTablet for tablet 9695e153d2f54b008f8c7201463cfbc8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251214 02:58:27.376416 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9695e153d2f54b008f8c7201463cfbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.375061 21329 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:27.380530 21323 ts_tablet_manager.cc:1434] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.075s user 0.000s sys 0.009s
I20251214 02:58:27.380909 21189 tablet_service.cc:1505] Processing CreateTablet for tablet 75e354ca56944ff1b7775a8309bca568 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251214 02:58:27.381630 21323 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:27.384243 21269 tablet_service.cc:1505] Processing CreateTablet for tablet 5751534b61ed4568abd62057238d5fda (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251214 02:58:27.385603 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751534b61ed4568abd62057238d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.386716 21119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce7181915c14c369bac4b962d34913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.388285 21322 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:27.388998 21193 tablet_service.cc:1505] Processing CreateTablet for tablet 9fdf7633f868447a8abd2adfe9b04fc5 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251214 02:58:27.378441 21329 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.390820 21267 tablet_service.cc:1505] Processing CreateTablet for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251214 02:58:27.392118 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.395701 21327 ts_tablet_manager.cc:1434] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.339s user 0.004s sys 0.007s
I20251214 02:58:27.396183 21270 tablet_service.cc:1505] Processing CreateTablet for tablet 0c25ad2d7ede4ddeb75d46946114529b (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251214 02:58:27.406482 21329 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:27.407249 21189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e354ca56944ff1b7775a8309bca568. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.407176 21268 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd36aadbf4841c9a26e48e96430c009 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251214 02:58:27.407948 21263 tablet_service.cc:1505] Processing CreateTablet for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251214 02:58:27.408615 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd36aadbf4841c9a26e48e96430c009. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.399433 21114 tablet_service.cc:1505] Processing CreateTablet for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251214 02:58:27.430932 21198 tablet_service.cc:1505] Processing CreateTablet for tablet 5751534b61ed4568abd62057238d5fda (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251214 02:58:27.431495 21196 tablet_service.cc:1505] Processing CreateTablet for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251214 02:58:27.434736 21264 tablet_service.cc:1505] Processing CreateTablet for tablet 320811ca9d184da38a1197ebfdb7cd0f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251214 02:58:27.443553 21323 tablet_bootstrap.cc:654] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.436529 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.437216 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.454743 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdf7633f868447a8abd2adfe9b04fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.455173 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 9695e153d2f54b008f8c7201463cfbc8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251214 02:58:27.455488 21122 tablet_service.cc:1505] Processing CreateTablet for tablet 9695e153d2f54b008f8c7201463cfbc8 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251214 02:58:27.456491 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9695e153d2f54b008f8c7201463cfbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.465977 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751534b61ed4568abd62057238d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.467061 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 0c25ad2d7ede4ddeb75d46946114529b (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251214 02:58:27.468322 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c25ad2d7ede4ddeb75d46946114529b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.468187 21115 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd36aadbf4841c9a26e48e96430c009 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251214 02:58:27.464818 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdf7633f868447a8abd2adfe9b04fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.464748 21264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320811ca9d184da38a1197ebfdb7cd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.470224 21115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd36aadbf4841c9a26e48e96430c009. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.470902 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.471858 21327 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:27.471895 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5751534b61ed4568abd62057238d5fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.476958 21329 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:27.477442 21329 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.479492 21329 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:27.476666 21120 tablet_service.cc:1505] Processing CreateTablet for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251214 02:58:27.486764 21322 tablet_bootstrap.cc:654] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.487051 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.487236 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37ffa0f0d64c3c8b5f77f26ff5f76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.489121 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9695e153d2f54b008f8c7201463cfbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.452009 21191 tablet_service.cc:1505] Processing CreateTablet for tablet 0c25ad2d7ede4ddeb75d46946114529b (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251214 02:58:27.500077 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c25ad2d7ede4ddeb75d46946114529b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.514652 21197 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd36aadbf4841c9a26e48e96430c009 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251214 02:58:27.517009 21116 tablet_service.cc:1505] Processing CreateTablet for tablet 320811ca9d184da38a1197ebfdb7cd0f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251214 02:58:27.521555 21123 tablet_service.cc:1505] Processing CreateTablet for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251214 02:58:27.524081 21192 tablet_service.cc:1505] Processing CreateTablet for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251214 02:58:27.528407 21195 tablet_service.cc:1505] Processing CreateTablet for tablet 320811ca9d184da38a1197ebfdb7cd0f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251214 02:58:27.533051 21327 tablet_bootstrap.cc:654] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.534950 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd36aadbf4841c9a26e48e96430c009. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.518406 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320811ca9d184da38a1197ebfdb7cd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.539036 21269 tablet_service.cc:1505] Processing CreateTablet for tablet e1dc028308e3434791fdd7169a0858b0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251214 02:58:27.556301 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dc028308e3434791fdd7169a0858b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.552695 21189 tablet_service.cc:1505] Processing CreateTablet for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251214 02:58:27.567029 21328 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:27.572516 21328 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.573647 21323 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:27.575524 21328 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:27.577031 21190 tablet_service.cc:1505] Processing CreateTablet for tablet e1dc028308e3434791fdd7169a0858b0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251214 02:58:27.574257 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.581233 21266 tablet_service.cc:1505] Processing CreateTablet for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251214 02:58:27.582592 21266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.582774 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:27.584574 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:27.586206 21173 leader_election.cc:343] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:27.586915 21173 leader_election.cc:343] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:27.593200 21192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.590386 21323 ts_tablet_manager.cc:1403] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.209s user 0.015s sys 0.001s
I20251214 02:58:27.598896 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c25ad2d7ede4ddeb75d46946114529b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.606911 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320811ca9d184da38a1197ebfdb7cd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.609294 21189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.610872 21272 tablet_service.cc:1505] Processing CreateTablet for tablet 5ce8bd94962e4bf48e01e9e747adba15 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251214 02:58:27.609349 21270 tablet_service.cc:1505] Processing CreateTablet for tablet 8cd37022763049b2829947b4a33d6aea (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251214 02:58:27.613842 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:27.615127 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:27.616119 21268 tablet_service.cc:1505] Processing CreateTablet for tablet 5bdfefe7cfb64f5cba716a427e83fc9f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
W20251214 02:58:27.617126 21247 leader_election.cc:343] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:27.617870 21247 leader_election.cc:343] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:27.622290 21323 raft_consensus.cc:359] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.623071 21323 raft_consensus.cc:385] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.623268 21322 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:27.623937 21322 ts_tablet_manager.cc:1403] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.236s user 0.008s sys 0.010s
I20251214 02:58:27.628230 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dc028308e3434791fdd7169a0858b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.612150 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce8bd94962e4bf48e01e9e747adba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.630379 21265 tablet_service.cc:1505] Processing CreateTablet for tablet ae2c1119874240b6af01af115f82ecd3 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251214 02:58:27.635957 21264 tablet_service.cc:1505] Processing CreateTablet for tablet 205cf8366aa44b27a349916da03cff46 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251214 02:58:27.637013 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c1119874240b6af01af115f82ecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.639132 21271 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca27c31805c4e55898fe4f60468ffa6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251214 02:58:27.640343 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca27c31805c4e55898fe4f60468ffa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.641875 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdfefe7cfb64f5cba716a427e83fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.641949 21267 tablet_service.cc:1505] Processing CreateTablet for tablet 172b9f2bc43b42fca62c756aa5cf17a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251214 02:58:27.643167 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 172b9f2bc43b42fca62c756aa5cf17a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.646881 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd37022763049b2829947b4a33d6aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.647115 21114 tablet_service.cc:1505] Processing CreateTablet for tablet e1dc028308e3434791fdd7169a0858b0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251214 02:58:27.648320 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dc028308e3434791fdd7169a0858b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.651815 21264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205cf8366aa44b27a349916da03cff46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.658005 21117 tablet_service.cc:1505] Processing CreateTablet for tablet 8cd37022763049b2829947b4a33d6aea (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251214 02:58:27.660010 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd37022763049b2829947b4a33d6aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.661006 21121 tablet_service.cc:1505] Processing CreateTablet for tablet 5ce8bd94962e4bf48e01e9e747adba15 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251214 02:58:27.662228 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce8bd94962e4bf48e01e9e747adba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.623361 21323 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.663174 21323 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.663985 21119 tablet_service.cc:1505] Processing CreateTablet for tablet 5bdfefe7cfb64f5cba716a427e83fc9f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251214 02:58:27.664188 21122 tablet_service.cc:1505] Processing CreateTablet for tablet ae2c1119874240b6af01af115f82ecd3 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251214 02:58:27.665179 21119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdfefe7cfb64f5cba716a427e83fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.667112 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c1119874240b6af01af115f82ecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.671401 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 205cf8366aa44b27a349916da03cff46 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251214 02:58:27.672730 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205cf8366aa44b27a349916da03cff46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.678236 21115 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca27c31805c4e55898fe4f60468ffa6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251214 02:58:27.679538 21115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca27c31805c4e55898fe4f60468ffa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.684417 21328 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:27.684947 21328 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.687067 21328 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:27.688879 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:27.689580 21133 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:27.690881 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:27.691915 21328 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:27.692288 21328 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:27.692612 21328 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:27.719755 21323 ts_tablet_manager.cc:1434] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.121s user 0.004s sys 0.004s
I20251214 02:58:27.721073 21323 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:27.725533 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:27.726260 21282 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:27.726764 21196 tablet_service.cc:1505] Processing CreateTablet for tablet 5ce8bd94962e4bf48e01e9e747adba15 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251214 02:58:27.687371 21193 tablet_service.cc:1505] Processing CreateTablet for tablet 8cd37022763049b2829947b4a33d6aea (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251214 02:58:27.727594 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:27.728835 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:27.730792 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:27.743203 21328 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.744418 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd37022763049b2829947b4a33d6aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.744545 21189 tablet_service.cc:1505] Processing CreateTablet for tablet 5bdfefe7cfb64f5cba716a427e83fc9f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251214 02:58:27.746868 21328 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:27.747455 21328 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:27.747788 21328 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:27.753480 21263 tablet_service.cc:1505] Processing CreateTablet for tablet 19fd2760f37a43e2983b015db150e285 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251214 02:58:27.754148 21120 tablet_service.cc:1505] Processing CreateTablet for tablet 172b9f2bc43b42fca62c756aa5cf17a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251214 02:58:27.754813 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fd2760f37a43e2983b015db150e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.755395 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 172b9f2bc43b42fca62c756aa5cf17a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.748148 21328 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:27.771106 21189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdfefe7cfb64f5cba716a427e83fc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.772466 21117 tablet_service.cc:1505] Processing CreateTablet for tablet 278f5a62c6a74101ae76cfc7325bcca0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251214 02:58:27.773783 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f5a62c6a74101ae76cfc7325bcca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.775928 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:27.777267 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:27.779141 21248 leader_election.cc:343] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:27.779603 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:27.780772 21329 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:27.722795 21322 raft_consensus.cc:359] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.787037 21322 raft_consensus.cc:385] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.787389 21322 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.788156 21322 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.789161 21281 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:27.792232 21322 ts_tablet_manager.cc:1434] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.168s user 0.005s sys 0.001s
I20251214 02:58:27.807032 21198 tablet_service.cc:1505] Processing CreateTablet for tablet ae2c1119874240b6af01af115f82ecd3 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251214 02:58:27.808302 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c1119874240b6af01af115f82ecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.811712 21122 tablet_service.cc:1505] Processing CreateTablet for tablet f0e59ca14c1548a686e85b1a9fcf9288 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251214 02:58:27.812932 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e59ca14c1548a686e85b1a9fcf9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.814950 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce8bd94962e4bf48e01e9e747adba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.816457 21264 tablet_service.cc:1505] Processing CreateTablet for tablet 278f5a62c6a74101ae76cfc7325bcca0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251214 02:58:27.817641 21264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f5a62c6a74101ae76cfc7325bcca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.819890 21190 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca27c31805c4e55898fe4f60468ffa6 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251214 02:58:27.821079 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca27c31805c4e55898fe4f60468ffa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.823197 21271 tablet_service.cc:1505] Processing CreateTablet for tablet f0e59ca14c1548a686e85b1a9fcf9288 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251214 02:58:27.824343 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e59ca14c1548a686e85b1a9fcf9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.830690 21323 tablet_bootstrap.cc:654] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.832690 21116 tablet_service.cc:1505] Processing CreateTablet for tablet 755b818717824383985748e28244d7a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251214 02:58:27.833880 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755b818717824383985748e28244d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.834529 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 172b9f2bc43b42fca62c756aa5cf17a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251214 02:58:27.844393 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 172b9f2bc43b42fca62c756aa5cf17a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.847167 21265 tablet_service.cc:1505] Processing CreateTablet for tablet 4b611d0c068c451bb4eeb7eda4823036 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251214 02:58:27.848378 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b611d0c068c451bb4eeb7eda4823036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.858928 21115 tablet_service.cc:1505] Processing CreateTablet for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251214 02:58:27.864535 21197 tablet_service.cc:1505] Processing CreateTablet for tablet 19fd2760f37a43e2983b015db150e285 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251214 02:58:27.865963 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fd2760f37a43e2983b015db150e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.843860 21263 tablet_service.cc:1505] Processing CreateTablet for tablet 755b818717824383985748e28244d7a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251214 02:58:27.867479 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755b818717824383985748e28244d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.874154 21322 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:27.875008 21191 tablet_service.cc:1505] Processing CreateTablet for tablet 278f5a62c6a74101ae76cfc7325bcca0 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251214 02:58:27.879448 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f5a62c6a74101ae76cfc7325bcca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.879506 21123 tablet_service.cc:1505] Processing CreateTablet for tablet 19fd2760f37a43e2983b015db150e285 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251214 02:58:27.887423 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fd2760f37a43e2983b015db150e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.887236 21267 tablet_service.cc:1505] Processing CreateTablet for tablet b98a871e4a86480da87fed8511a68575 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251214 02:58:27.888615 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a871e4a86480da87fed8511a68575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.884373 21328 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:27.889735 21207 raft_consensus.cc:2393] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:27.895143 21192 tablet_service.cc:1505] Processing CreateTablet for tablet f0e59ca14c1548a686e85b1a9fcf9288 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251214 02:58:27.895349 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:27.885762 21195 tablet_service.cc:1505] Processing CreateTablet for tablet 205cf8366aa44b27a349916da03cff46 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251214 02:58:27.896415 21329 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:27.897300 21270 tablet_service.cc:1505] Processing CreateTablet for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251214 02:58:27.898023 21272 tablet_service.cc:1505] Processing CreateTablet for tablet a2f02493ca3c4039b61ed09e00355cc2 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251214 02:58:27.898730 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.903131 21329 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:27.905211 21269 tablet_service.cc:1505] Processing CreateTablet for tablet a54bd91b04344ee0b499fb117cc9fb91 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251214 02:58:27.906450 21269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54bd91b04344ee0b499fb117cc9fb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.907967 21121 tablet_service.cc:1505] Processing CreateTablet for tablet b98a871e4a86480da87fed8511a68575 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251214 02:58:27.914582 21268 tablet_service.cc:1505] Processing CreateTablet for tablet 3dcb97eb0c4e40bbabc681854107bb92 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251214 02:58:27.916996 21327 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:27.917553 21327 ts_tablet_manager.cc:1403] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.446s user 0.047s sys 0.074s
I20251214 02:58:27.865046 21114 tablet_service.cc:1505] Processing CreateTablet for tablet 4b611d0c068c451bb4eeb7eda4823036 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251214 02:58:27.916038 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcb97eb0c4e40bbabc681854107bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.924346 21193 tablet_service.cc:1505] Processing CreateTablet for tablet 4b611d0c068c451bb4eeb7eda4823036 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251214 02:58:27.926304 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b611d0c068c451bb4eeb7eda4823036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.926944 21192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e59ca14c1548a686e85b1a9fcf9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.927750 21121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a871e4a86480da87fed8511a68575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.928313 21115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.936444 21322 tablet_bootstrap.cc:654] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:27.936648 21119 tablet_service.cc:1505] Processing CreateTablet for tablet a2f02493ca3c4039b61ed09e00355cc2 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251214 02:58:27.938099 21119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f02493ca3c4039b61ed09e00355cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.946827 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205cf8366aa44b27a349916da03cff46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.949209 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b611d0c068c451bb4eeb7eda4823036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.956857 21322 tablet_bootstrap.cc:492] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:27.957341 21322 ts_tablet_manager.cc:1403] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.008s
I20251214 02:58:27.924228 21327 raft_consensus.cc:359] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.959308 21327 raft_consensus.cc:385] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.959621 21327 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.960033 21322 raft_consensus.cc:359] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.960724 21322 raft_consensus.cc:385] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:27.960386 21327 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:27.961030 21322 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:27.961787 21322 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:27.965075 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 234428us
W20251214 02:58:27.965911 21173 leader_election.cc:343] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:27.914582 21266 tablet_service.cc:1505] Processing CreateTablet for tablet 434bbcb8330a4ba680c1346b33460471 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251214 02:58:27.967104 21322 ts_tablet_manager.cc:1434] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20251214 02:58:27.966986 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:27.968148 21322 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:27.974938 21328 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:27.975504 21328 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:27.975865 21328 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:27.976186 21328 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:27.979182 21323 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:27.985608 21198 tablet_service.cc:1505] Processing CreateTablet for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251214 02:58:27.979640 21323 ts_tablet_manager.cc:1403] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.259s user 0.010s sys 0.005s
I20251214 02:58:27.989883 21116 tablet_service.cc:1505] Processing CreateTablet for tablet a54bd91b04344ee0b499fb117cc9fb91 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251214 02:58:27.970615 21272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f02493ca3c4039b61ed09e00355cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:27.992322 21120 tablet_service.cc:1505] Processing CreateTablet for tablet 3dcb97eb0c4e40bbabc681854107bb92 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251214 02:58:27.992322 21117 tablet_service.cc:1505] Processing CreateTablet for tablet 434bbcb8330a4ba680c1346b33460471 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251214 02:58:28.002753 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.008507 21194 tablet_service.cc:1505] Processing CreateTablet for tablet 755b818717824383985748e28244d7a1 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251214 02:58:28.029112 21133 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.029738 21194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755b818717824383985748e28244d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.030910 21266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 434bbcb8330a4ba680c1346b33460471. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.031148 21265 tablet_service.cc:1505] Processing CreateTablet for tablet 01028fc01cbf4f4c88f7e36696d08d03 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251214 02:58:28.032339 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01028fc01cbf4f4c88f7e36696d08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.004235 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.033965 21281 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.004860 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.006217 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.009971 21196 tablet_service.cc:1505] Processing CreateTablet for tablet b98a871e4a86480da87fed8511a68575 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251214 02:58:28.039072 21133 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
W20251214 02:58:28.040493 21173 leader_election.cc:343] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.041270 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.011816 21264 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba6dea370cb421894536a9154735510 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251214 02:58:28.002269 21263 tablet_service.cc:1505] Processing CreateTablet for tablet e18f39bed60f44f5b8c25ede5d8bba8f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251214 02:58:28.020991 21327 ts_tablet_manager.cc:1434] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.103s user 0.007s sys 0.004s
I20251214 02:58:28.021324 21192 tablet_service.cc:1505] Processing CreateTablet for tablet a2f02493ca3c4039b61ed09e00355cc2 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251214 02:58:28.023401 21328 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.025646 21323 raft_consensus.cc:359] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.042218 21344 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.045545 21323 raft_consensus.cc:385] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.051904 21323 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.053071 21323 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.060328 21118 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba6dea370cb421894536a9154735510 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251214 02:58:28.063911 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.064222 21196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98a871e4a86480da87fed8511a68575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.064503 21327 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.064539 21282 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.055807 21328 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.057241 21117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 434bbcb8330a4ba680c1346b33460471. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.065449 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.058782 21264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba6dea370cb421894536a9154735510. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.070863 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18f39bed60f44f5b8c25ede5d8bba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.059264 21281 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:28.071395 21116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54bd91b04344ee0b499fb117cc9fb91. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 02:58:28.072654 21173 leader_election.cc:343] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.073536 21120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcb97eb0c4e40bbabc681854107bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.057523 21280 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:28.064179 21123 tablet_service.cc:1505] Processing CreateTablet for tablet e18f39bed60f44f5b8c25ede5d8bba8f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251214 02:58:28.080595 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18f39bed60f44f5b8c25ede5d8bba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.066970 21192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f02493ca3c4039b61ed09e00355cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.081434 21191 tablet_service.cc:1505] Processing CreateTablet for tablet a54bd91b04344ee0b499fb117cc9fb91 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251214 02:58:28.088862 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54bd91b04344ee0b499fb117cc9fb91. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 02:58:28.091033 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:28.098807 21118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba6dea370cb421894536a9154735510. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.106926 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.093612 21322 tablet_bootstrap.cc:654] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.107074 21122 tablet_service.cc:1505] Processing CreateTablet for tablet 01028fc01cbf4f4c88f7e36696d08d03 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251214 02:58:28.107584 21328 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.107998 21328 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:28.108366 21122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01028fc01cbf4f4c88f7e36696d08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.108721 21328 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:28.102083 21271 tablet_service.cc:1505] Processing CreateTablet for tablet d5a50d11dd114861bc3ebe1138e3d999 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251214 02:58:28.119611 21271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a50d11dd114861bc3ebe1138e3d999. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.128743 21114 tablet_service.cc:1505] Processing CreateTablet for tablet d5a50d11dd114861bc3ebe1138e3d999 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251214 02:58:28.130105 21114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a50d11dd114861bc3ebe1138e3d999. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.143203 21282 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:28.146946 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.148324 21195 tablet_service.cc:1505] Processing CreateTablet for tablet 3dcb97eb0c4e40bbabc681854107bb92 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251214 02:58:28.150885 21193 tablet_service.cc:1505] Processing CreateTablet for tablet 434bbcb8330a4ba680c1346b33460471 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251214 02:58:28.150853 21198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.152688 21189 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba6dea370cb421894536a9154735510 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251214 02:58:28.153936 21189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba6dea370cb421894536a9154735510. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.156201 21197 tablet_service.cc:1505] Processing CreateTablet for tablet e18f39bed60f44f5b8c25ede5d8bba8f (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251214 02:58:28.157426 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18f39bed60f44f5b8c25ede5d8bba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.160048 21190 tablet_service.cc:1505] Processing CreateTablet for tablet 01028fc01cbf4f4c88f7e36696d08d03 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251214 02:58:28.183355 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 434bbcb8330a4ba680c1346b33460471. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.184593 21191 tablet_service.cc:1505] Processing CreateTablet for tablet d5a50d11dd114861bc3ebe1138e3d999 (DEFAULT_TABLE table=test-workload [id=89b54530dd4e42b7bc6235879c56c644]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251214 02:58:28.185886 21191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a50d11dd114861bc3ebe1138e3d999. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.195847 21344 raft_consensus.cc:697] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:28.199994 21195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcb97eb0c4e40bbabc681854107bb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.214149 21327 tablet_bootstrap.cc:654] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.203742 21328 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.230185 21347 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.229943 21344 consensus_queue.cc:237] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.230722 21347 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.232757 21328 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:28.233136 21347 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.233614 21328 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.249441 21323 ts_tablet_manager.cc:1434] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.232s user 0.007s sys 0.024s
I20251214 02:58:28.254176 21190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01028fc01cbf4f4c88f7e36696d08d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:28.287807 21323 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.300967 21322 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.301501 21322 ts_tablet_manager.cc:1403] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.334s user 0.011s sys 0.013s
I20251214 02:58:28.304473 21328 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.304988 21328 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.306947 21328 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.308254 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:28.309782 21098 leader_election.cc:343] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.309777 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:28.311513 21173 leader_election.cc:343] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.312390 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:28.313668 21173 leader_election.cc:343] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.314091 21173 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.315119 21328 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.304503 21322 raft_consensus.cc:359] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.317430 21322 raft_consensus.cc:385] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.317700 21322 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.319132 21322 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.320107 21327 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.320540 21327 ts_tablet_manager.cc:1403] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.258s user 0.010s sys 0.009s
I20251214 02:58:28.321420 21322 ts_tablet_manager.cc:1434] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20251214 02:58:28.311509 21323 tablet_bootstrap.cc:654] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.319432 21030 catalog_manager.cc:5654] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:28.323083 21327 raft_consensus.cc:359] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.323764 21327 raft_consensus.cc:385] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.324069 21327 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.324895 21327 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.327308 21322 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.329523 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:28.331151 21099 leader_election.cc:343] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.331629 21099 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:28.335012 21347 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.340060 21327 ts_tablet_manager.cc:1434] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20251214 02:58:28.341825 21327 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.347999 21327 tablet_bootstrap.cc:654] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.354929 21323 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.355432 21323 ts_tablet_manager.cc:1403] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.068s user 0.028s sys 0.007s
I20251214 02:58:28.359884 21322 tablet_bootstrap.cc:654] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.367806 21327 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.358251 21323 raft_consensus.cc:359] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.368460 21323 raft_consensus.cc:385] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.368831 21323 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.368474 21327 ts_tablet_manager.cc:1403] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20251214 02:58:28.372911 21327 raft_consensus.cc:359] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.373569 21327 raft_consensus.cc:385] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.373857 21327 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.374717 21322 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.375211 21322 ts_tablet_manager.cc:1403] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.010s
W20251214 02:58:28.375831 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 44880.9us
I20251214 02:58:28.376812 21323 consensus_queue.cc:260] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.377964 21322 raft_consensus.cc:359] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.378726 21322 raft_consensus.cc:385] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.379019 21322 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.379340 21323 ts_tablet_manager.cc:1434] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20251214 02:58:28.379791 21322 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.380362 21323 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.374495 21327 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.383396 21327 ts_tablet_manager.cc:1434] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20251214 02:58:28.389499 21322 ts_tablet_manager.cc:1434] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20251214 02:58:28.389900 21323 tablet_bootstrap.cc:654] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.393163 21327 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.397524 21322 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.398350 21031 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:28.399757 21327 tablet_bootstrap.cc:654] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.400590 21031 catalog_manager.cc:5654] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:28.404121 21322 tablet_bootstrap.cc:654] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.409617 21323 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.410115 21323 ts_tablet_manager.cc:1403] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.011s
I20251214 02:58:28.411322 21322 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.411784 21322 ts_tablet_manager.cc:1403] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20251214 02:58:28.412849 21323 raft_consensus.cc:359] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.413560 21323 raft_consensus.cc:385] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.413861 21323 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.414426 21322 raft_consensus.cc:359] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.414587 21323 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.415194 21322 raft_consensus.cc:385] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.415498 21322 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.416299 21322 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.418133 21327 tablet_bootstrap.cc:492] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.418591 21327 ts_tablet_manager.cc:1403] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.005s
I20251214 02:58:28.421244 21327 raft_consensus.cc:359] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.421906 21327 raft_consensus.cc:385] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.422190 21327 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.423228 21322 ts_tablet_manager.cc:1434] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251214 02:58:28.424072 21323 ts_tablet_manager.cc:1434] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20251214 02:58:28.424304 21322 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.422839 21327 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.427970 21323 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.430948 21322 tablet_bootstrap.cc:654] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.434516 21323 tablet_bootstrap.cc:654] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.439128 21327 ts_tablet_manager.cc:1434] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251214 02:58:28.440239 21327 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.446583 21327 tablet_bootstrap.cc:654] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.452490 21323 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.453024 21323 ts_tablet_manager.cc:1403] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.001s
I20251214 02:58:28.455889 21323 raft_consensus.cc:359] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.456629 21323 raft_consensus.cc:385] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.456928 21323 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.460888 21322 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.461354 21322 ts_tablet_manager.cc:1403] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.007s
I20251214 02:58:28.461508 21327 tablet_bootstrap.cc:492] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.462074 21327 ts_tablet_manager.cc:1403] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20251214 02:58:28.463719 21322 raft_consensus.cc:359] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.464376 21322 raft_consensus.cc:385] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.464644 21322 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.464936 21327 raft_consensus.cc:359] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.465248 21322 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.465631 21327 raft_consensus.cc:385] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.465950 21327 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.466609 21327 consensus_queue.cc:260] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.467444 21322 ts_tablet_manager.cc:1434] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251214 02:58:28.468604 21322 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.468829 21327 ts_tablet_manager.cc:1434] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:28.470278 21323 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.469803 21327 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.473841 21322 tablet_bootstrap.cc:654] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.473821 21323 ts_tablet_manager.cc:1434] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20251214 02:58:28.479450 21323 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.480098 21327 tablet_bootstrap.cc:654] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.488504 21329 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.489197 21323 tablet_bootstrap.cc:654] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.488999 21329 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.491004 21329 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.492637 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.493317 21133 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.495108 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.495741 21207 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.496111 21322 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.496517 21322 ts_tablet_manager.cc:1403] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.004s
I20251214 02:58:28.497522 21247 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.502678 21329 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.503077 21329 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.503391 21329 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.503386 21375 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.510919 21322 raft_consensus.cc:359] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.511859 21322 raft_consensus.cc:385] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.512171 21322 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.512882 21322 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.514775 21329 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.517426 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.518019 21133 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.520951 21327 tablet_bootstrap.cc:492] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.521574 21327 ts_tablet_manager.cc:1403] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.013s
I20251214 02:58:28.524919 21133 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:28.524678 21327 raft_consensus.cc:359] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:28.525760 21327 raft_consensus.cc:385] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.526139 21327 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.526085 21247 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.527590 21327 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:28.530021 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:28.530642 21207 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.536154 21327 ts_tablet_manager.cc:1434] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20251214 02:58:28.511109 21375 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.539541 21327 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.540817 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:28.542039 21247 leader_election.cc:343] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.544512 21360 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.545114 21207 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:28.544996 21360 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.546732 21378 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.547106 21360 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.547205 21378 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:28.548169 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.548089 21378 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.549247 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.549855 21208 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:28.550858 21099 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.552410 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:28.553696 21248 leader_election.cc:343] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.554124 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.548802 21282 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:28.556895 21360 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.557267 21360 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.557574 21360 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.557667 21379 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.559947 21327 tablet_bootstrap.cc:654] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.564857 21360 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.567610 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:28.568154 21208 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.572917 21029 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:28.574673 21375 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.577123 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.577715 21282 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.578964 21329 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.580670 21208 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:28.581532 21360 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.583384 21099 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.587797 21344 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:28.588685 21360 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.589370 21360 raft_consensus.cc:697] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:28.590189 21360 consensus_queue.cc:237] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.594370 21282 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:28.594961 21322 ts_tablet_manager.cc:1434] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.098s user 0.025s sys 0.015s
I20251214 02:58:28.596988 21327 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.597401 21327 ts_tablet_manager.cc:1403] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.002s
I20251214 02:58:28.598058 21323 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.598457 21323 ts_tablet_manager.cc:1403] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.119s user 0.022s sys 0.039s
I20251214 02:58:28.599860 21327 raft_consensus.cc:359] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.600486 21327 raft_consensus.cc:385] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.600739 21327 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.600848 21323 raft_consensus.cc:359] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.601724 21323 raft_consensus.cc:385] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.601363 21327 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.602000 21323 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.602557 21322 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.602743 21323 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.603667 21327 ts_tablet_manager.cc:1434] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251214 02:58:28.604555 21327 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.605110 21323 ts_tablet_manager.cc:1434] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251214 02:58:28.606326 21323 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.610031 21327 tablet_bootstrap.cc:654] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.611692 21323 tablet_bootstrap.cc:654] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.615270 21133 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.625640 21031 catalog_manager.cc:5654] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:28.629439 21323 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.629904 21323 ts_tablet_manager.cc:1403] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.003s
I20251214 02:58:28.632278 21323 raft_consensus.cc:359] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.632947 21323 raft_consensus.cc:385] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.633224 21323 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.633845 21323 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.636235 21322 tablet_bootstrap.cc:654] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.642239 21323 ts_tablet_manager.cc:1434] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251214 02:58:28.643239 21323 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.648624 21323 tablet_bootstrap.cc:654] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.655129 21349 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:28.658100 21327 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.658599 21327 ts_tablet_manager.cc:1403] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.027s
I20251214 02:58:28.661603 21377 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.662123 21377 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.662447 21349 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:28.664183 21322 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.664672 21322 ts_tablet_manager.cc:1403] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.002s
I20251214 02:58:28.665125 21327 raft_consensus.cc:359] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.665786 21327 raft_consensus.cc:385] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.666051 21327 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.666713 21327 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.667121 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.667724 21322 raft_consensus.cc:359] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.667771 21131 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.668800 21327 ts_tablet_manager.cc:1434] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251214 02:58:28.669872 21327 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.669641 21247 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.670493 21329 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.670897 21329 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.671217 21329 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.671670 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.672263 21208 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.674320 21377 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.676162 21327 tablet_bootstrap.cc:654] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.679055 21329 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.681789 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.682267 21131 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.691246 21348 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:28.696252 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:28.696923 21208 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.698081 21347 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.698652 21347 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.700831 21347 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.702035 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.703377 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.703573 21347 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.703404 21281 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:28.704205 21347 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.706055 21347 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.706501 21347 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.706758 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.706980 21347 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.708122 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:28.708803 21347 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.709285 21347 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.710889 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.711081 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:28.712472 21098 leader_election.cc:343] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:28.712730 21099 leader_election.cc:343] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.713155 21329 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.670258 21322 raft_consensus.cc:385] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.713677 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.714776 21131 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
W20251214 02:58:28.715143 21099 leader_election.cc:343] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.715601 21099 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:28.715862 21247 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.716336 21360 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.713801 21322 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.717518 21322 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.718429 21281 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:28.721738 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:28.722651 21360 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.723019 21360 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.723338 21360 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.731673 21360 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.733233 21377 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.734714 21377 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:28.734634 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:28.735147 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.735754 21281 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.735657 21377 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:28.736104 21099 leader_election.cc:343] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.747056 21360 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.752279 21281 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:28.752715 21323 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.753149 21323 ts_tablet_manager.cc:1403] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.110s user 0.015s sys 0.007s
I20251214 02:58:28.753443 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:28.754534 21360 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.755010 21360 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:28.755741 21323 raft_consensus.cc:359] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.756395 21323 raft_consensus.cc:385] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.760217 21323 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.760890 21323 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.763199 21323 ts_tablet_manager.cc:1434] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20251214 02:58:28.765799 21323 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.755831 21360 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.778553 21030 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:28.783015 21347 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.790104 21208 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:28.790283 21328 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.794337 21328 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.796042 21348 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:28.796448 21328 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.800155 21348 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.800462 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.800629 21348 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:28.801789 21173 leader_election.cc:343] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.802551 21348 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.799638 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:28.805341 21173 leader_election.cc:343] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:28.805814 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.806641 21348 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.793255 21322 ts_tablet_manager.cc:1434] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.128s user 0.009s sys 0.000s
I20251214 02:58:28.808290 21322 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.793891 21323 tablet_bootstrap.cc:654] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.810979 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.811821 21280 raft_consensus.cc:2393] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:28.812745 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.814217 21173 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:28.815079 21348 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.822073 21348 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:28.822783 21031 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:28.823741 21322 tablet_bootstrap.cc:654] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.825407 21323 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.825800 21323 ts_tablet_manager.cc:1403] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.000s
I20251214 02:58:28.828174 21323 raft_consensus.cc:359] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.828827 21323 raft_consensus.cc:385] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.829102 21323 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.829828 21323 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.831318 21348 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.831797 21348 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.833753 21348 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.835625 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.836247 21282 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:28.837518 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:28.838330 21348 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.838673 21348 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.838989 21348 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.839946 21327 tablet_bootstrap.cc:492] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.841362 21327 ts_tablet_manager.cc:1403] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.171s user 0.076s sys 0.041s
I20251214 02:58:28.843631 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.847666 21377 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.848145 21377 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.848989 21323 ts_tablet_manager.cc:1434] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20251214 02:58:28.849714 21323 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:28.849996 21377 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.846477 21344 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:28.850950 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.843650 21327 raft_consensus.cc:359] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.853320 21327 raft_consensus.cc:385] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.853570 21327 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.854247 21327 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.855772 21132 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:28.856564 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:28.857087 21323 tablet_bootstrap.cc:654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.857142 21208 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.847198 21348 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.859860 21348 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.860935 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.861517 21132 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.863668 21248 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:28.864748 21377 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.865108 21377 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.865419 21377 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.868323 21322 tablet_bootstrap.cc:492] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.868821 21322 ts_tablet_manager.cc:1403] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.005s
I20251214 02:58:28.871263 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:28.872113 21280 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.871754 21322 raft_consensus.cc:359] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.872641 21322 raft_consensus.cc:385] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.872931 21322 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.874156 21322 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.876736 21322 ts_tablet_manager.cc:1434] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251214 02:58:28.877756 21322 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.881526 21327 ts_tablet_manager.cc:1434] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20251214 02:58:28.851614 21130 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:28.882593 21327 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.892916 21280 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:28.901288 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:28.896663 21377 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.902158 21348 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.902679 21348 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:28.903460 21348 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.904865 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:28.905496 21130 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.892590 21132 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:28.898118 21327 tablet_bootstrap.cc:654] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.895941 21322 tablet_bootstrap.cc:654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.918183 21029 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:28.922087 21130 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:28.923353 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.923676 21327 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.924090 21327 ts_tablet_manager.cc:1403] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.004s
I20251214 02:58:28.924194 21329 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:28.924649 21329 raft_consensus.cc:697] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:28.925467 21329 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.926693 21327 raft_consensus.cc:359] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.927322 21327 raft_consensus.cc:385] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.927593 21327 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.929621 21327 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.931725 21327 ts_tablet_manager.cc:1434] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251214 02:58:28.932683 21327 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:28.934684 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:28.935309 21208 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.936854 21322 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.937214 21322 ts_tablet_manager.cc:1403] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.005s
I20251214 02:58:28.939147 21322 raft_consensus.cc:359] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.939749 21322 raft_consensus.cc:385] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.940025 21322 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.940814 21322 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.942471 21327 tablet_bootstrap.cc:654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.943082 21322 ts_tablet_manager.cc:1434] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251214 02:58:28.943298 21208 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:28.944028 21322 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.949316 21377 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:28.950407 21322 tablet_bootstrap.cc:654] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.962157 21322 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:28.962685 21322 ts_tablet_manager.cc:1403] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20251214 02:58:28.965714 21322 raft_consensus.cc:359] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.965983 21329 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:28.966403 21322 raft_consensus.cc:385] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.966723 21322 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.961526 21031 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:28.970841 21322 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.973476 21322 ts_tablet_manager.cc:1434] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251214 02:58:28.974658 21322 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:28.982662 21348 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:28.983180 21348 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.984019 21322 tablet_bootstrap.cc:654] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:28.985601 21348 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:28.986732 21323 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:28.986796 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:28.987236 21323 ts_tablet_manager.cc:1403] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.138s user 0.070s sys 0.046s
I20251214 02:58:28.987325 21327 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:28.987459 21130 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:28.988058 21327 ts_tablet_manager.cc:1403] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.056s user 0.023s sys 0.020s
I20251214 02:58:28.988667 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:28.989480 21280 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:28.989836 21323 raft_consensus.cc:359] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.990455 21323 raft_consensus.cc:385] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:28.990763 21323 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:28.990633 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:28.992769 21348 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:28.993151 21348 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:28.993705 21348 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:28.996125 21323 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:28.997759 21323 ts_tablet_manager.cc:1434] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251214 02:58:29.000699 21323 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.004673 21348 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.005497 21347 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.005960 21347 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.008060 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:29.008751 21130 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.007233 21348 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.010999 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:29.011178 21327 raft_consensus.cc:359] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.011605 21280 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.011827 21327 raft_consensus.cc:385] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.012138 21327 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.012739 21327 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.014693 21327 ts_tablet_manager.cc:1434] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20251214 02:58:29.015529 21327 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.008095 21347 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.018642 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.022636 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:29.024899 21099 leader_election.cc:343] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.026631 21323 tablet_bootstrap.cc:654] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.027300 21327 tablet_bootstrap.cc:654] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.027040 21375 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 02:58:29.028654 21098 leader_election.cc:343] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.029120 21098 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:29.035044 21322 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.035542 21322 ts_tablet_manager.cc:1403] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.011s
I20251214 02:58:29.037998 21322 raft_consensus.cc:359] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.034942 21360 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.040016 21322 raft_consensus.cc:385] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.040303 21322 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.041004 21322 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.043251 21322 ts_tablet_manager.cc:1434] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251214 02:58:29.044091 21322 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.049288 21323 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.049851 21323 ts_tablet_manager.cc:1403] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.009s
I20251214 02:58:29.052480 21323 raft_consensus.cc:359] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.053150 21323 raft_consensus.cc:385] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.053467 21322 tablet_bootstrap.cc:654] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.053412 21323 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.054363 21323 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.060424 21323 ts_tablet_manager.cc:1434] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251214 02:58:29.061507 21323 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.067447 21323 tablet_bootstrap.cc:654] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.078783 21323 tablet_bootstrap.cc:492] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.079298 21323 ts_tablet_manager.cc:1403] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20251214 02:58:29.082043 21323 raft_consensus.cc:359] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.082774 21323 raft_consensus.cc:385] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.083047 21323 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.083796 21323 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.091107 21323 ts_tablet_manager.cc:1434] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20251214 02:58:29.092204 21323 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.102334 21323 tablet_bootstrap.cc:654] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:29.107841 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:29.111160 21130 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:29.112594 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:29.113487 21348 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:29.113981 21348 raft_consensus.cc:697] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:29.114925 21280 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:29.114814 21348 consensus_queue.cc:237] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.117909 21323 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.118399 21323 ts_tablet_manager.cc:1403] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.008s
I20251214 02:58:29.120971 21323 raft_consensus.cc:359] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.121721 21323 raft_consensus.cc:385] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.122051 21323 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.122924 21323 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.128288 21323 ts_tablet_manager.cc:1434] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20251214 02:58:29.129323 21323 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.135943 21323 tablet_bootstrap.cc:654] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.142046 21327 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.142496 21327 ts_tablet_manager.cc:1403] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.127s user 0.051s sys 0.032s
I20251214 02:58:29.143303 21322 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.143862 21322 ts_tablet_manager.cc:1403] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.100s user 0.044s sys 0.032s
I20251214 02:58:29.144948 21327 raft_consensus.cc:359] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.145653 21327 raft_consensus.cc:385] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.145917 21327 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.145398 21027 catalog_manager.cc:5654] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:29.146986 21322 raft_consensus.cc:359] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.147224 21327 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.147817 21322 raft_consensus.cc:385] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.148223 21322 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.149418 21327 ts_tablet_manager.cc:1434] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251214 02:58:29.149044 21322 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.150261 21327 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.151572 21322 ts_tablet_manager.cc:1434] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20251214 02:58:29.152786 21322 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.156780 21327 tablet_bootstrap.cc:654] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.159044 21322 tablet_bootstrap.cc:654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.162035 21348 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.162554 21348 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.163762 21347 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.164616 21348 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.167186 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.168185 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:29.169077 21173 leader_election.cc:343] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:29.169883 21173 leader_election.cc:343] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.170288 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.172833 21348 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.175199 21347 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.191762 21348 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.192291 21348 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.197433 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.198159 21281 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:29.199543 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:29.200465 21344 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:29.200832 21344 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:29.201148 21344 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.211313 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.212092 21132 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:29.194309 21348 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.223209 21375 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.223812 21323 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.223722 21375 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.224339 21323 ts_tablet_manager.cc:1403] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.095s user 0.038s sys 0.050s
I20251214 02:58:29.225641 21375 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:29.227873 21323 raft_consensus.cc:359] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.228415 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.229387 21208 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:29.228600 21323 raft_consensus.cc:385] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.230543 21323 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.230762 21248 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:29.231091 21327 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.231600 21327 ts_tablet_manager.cc:1403] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.082s user 0.033s sys 0.027s
I20251214 02:58:29.231199 21323 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.233385 21323 ts_tablet_manager.cc:1434] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20251214 02:58:29.234220 21323 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.234004 21327 raft_consensus.cc:359] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.234661 21327 raft_consensus.cc:385] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.234932 21327 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.235680 21327 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.236830 21375 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:29.237176 21375 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:29.237491 21347 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.238529 21327 ts_tablet_manager.cc:1434] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 02:58:29.239398 21327 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.237478 21375 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.240100 21323 tablet_bootstrap.cc:654] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.245925 21327 tablet_bootstrap.cc:654] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.247705 21322 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.248157 21322 ts_tablet_manager.cc:1403] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.096s user 0.015s sys 0.005s
I20251214 02:58:29.250504 21322 raft_consensus.cc:359] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.251166 21322 raft_consensus.cc:385] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.251410 21322 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.252416 21322 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.259328 21347 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.259850 21347 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.263079 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.264096 21322 ts_tablet_manager.cc:1434] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
W20251214 02:58:29.264320 21098 leader_election.cc:343] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.264843 21347 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.266808 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.267529 21132 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:29.272104 21347 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.274282 21349 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.274797 21349 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.276639 21349 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.277256 21322 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.281365 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.281999 21280 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:29.284792 21322 tablet_bootstrap.cc:654] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.286347 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:29.288298 21328 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:29.288691 21328 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:29.289012 21328 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.291188 21377 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.301528 21347 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.302136 21347 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.306293 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.307006 21280 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:29.308375 21098 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:29.309229 21360 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:29.309597 21360 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:29.309911 21360 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.319492 21377 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:29.332787 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:29.334769 21173 leader_election.cc:343] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.340409 21360 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.350953 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.362888 21377 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.386577 21377 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.390470 21377 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:29.392354 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:29.394217 21248 leader_election.cc:343] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.369251 21347 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.372797 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.395526 21204 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:29.396941 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:29.398422 21247 leader_election.cc:343] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.398916 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.400384 21377 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.404919 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:29.405602 21280 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:29.412320 21099 leader_election.cc:343] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.412900 21099 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:29.416178 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:29.416855 21204 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.425542 21204 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:29.426600 21347 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.440799 21280 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:29.366211 21344 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.444262 21329 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.446094 21099 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:29.534289 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 88330.2us
I20251214 02:58:29.536155 21397 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:29.537287 21378 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.447333 21360 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.449110 21375 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.488976 21348 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.507784 21349 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:29.554481 21348 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.557108 21348 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.508632 21377 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.381636 21328 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.568922 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.598783 21397 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:29.630748 21397 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.567276 21377 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:29.637542 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 102144us
I20251214 02:58:29.639348 21344 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:29.595531 21173 leader_election.cc:343] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:29.644047 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 53330.7us
I20251214 02:58:29.645098 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.651886 21377 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:29.657025 21344 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 02:58:29.660013 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.660211 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:29.660722 21133 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:29.661667 21248 leader_election.cc:343] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.662465 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:29.663061 21208 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.667583 21133 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:29.668745 21327 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
W20251214 02:58:29.669060 21173 leader_election.cc:343] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.669529 21173 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.670120 21344 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.672063 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.673514 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:29.675022 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:29.675618 21132 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.675899 21322 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.676368 21322 ts_tablet_manager.cc:1403] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.399s user 0.113s sys 0.120s
I20251214 02:58:29.669207 21327 ts_tablet_manager.cc:1403] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.430s user 0.130s sys 0.259s
I20251214 02:58:29.656603 21375 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:29.681850 21208 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:29.683228 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:29.683473 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:29.683804 21282 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.684170 21323 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.684621 21323 ts_tablet_manager.cc:1403] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.451s user 0.143s sys 0.269s
I20251214 02:58:29.685232 21280 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.687651 21327 raft_consensus.cc:359] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.658069 21328 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.689183 21327 raft_consensus.cc:385] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.690527 21327 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.691764 21327 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.673601 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:58:29.693739 21247 leader_election.cc:343] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.697623 21247 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.698583 21327 ts_tablet_manager.cc:1434] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20251214 02:58:29.698913 21432 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:29.699018 21375 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.699532 21327 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.699460 21432 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:29.700441 21432 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.702385 21323 raft_consensus.cc:359] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.707382 21323 raft_consensus.cc:385] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.707690 21323 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.714075 21323 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.751171 21322 raft_consensus.cc:359] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.751956 21322 raft_consensus.cc:385] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.752228 21322 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
W20251214 02:58:29.749421 21173 leader_election.cc:343] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.729660 21282 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:29.761601 21132 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:29.781885 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.784015 21349 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:29.784487 21349 raft_consensus.cc:697] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:29.785277 21349 consensus_queue.cc:237] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.797261 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:29.798388 21348 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:29.798920 21348 raft_consensus.cc:697] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:29.799790 21348 consensus_queue.cc:237] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.805752 21323 ts_tablet_manager.cc:1434] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.121s user 0.011s sys 0.003s
I20251214 02:58:29.807467 21323 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.713694 21328 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.095s
I20251214 02:58:29.809355 21344 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:29.837011 21327 tablet_bootstrap.cc:654] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.839740 21322 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.842530 21322 ts_tablet_manager.cc:1434] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.166s user 0.007s sys 0.007s
I20251214 02:58:29.846160 21322 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.809718 21027 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:29.843026 21280 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:29.857322 21348 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.859239 21348 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.862457 21323 tablet_bootstrap.cc:654] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.864687 21348 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.867924 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.870244 21322 tablet_bootstrap.cc:654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:29.870488 21173 leader_election.cc:343] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.869683 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.873476 21433 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:29.874900 21173 leader_election.cc:343] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.875327 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.876521 21327 tablet_bootstrap.cc:492] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.876950 21327 ts_tablet_manager.cc:1403] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.178s user 0.016s sys 0.008s
I20251214 02:58:29.877080 21328 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.873975 21433 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.881037 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.881664 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:29.882522 21248 leader_election.cc:343] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.883038 21433 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:29.882908 21327 raft_consensus.cc:359] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:29.883589 21247 leader_election.cc:343] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.884049 21247 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.883603 21327 raft_consensus.cc:385] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.885205 21433 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.885253 21327 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.886615 21327 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.889139 21327 ts_tablet_manager.cc:1434] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20251214 02:58:29.890012 21327 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.894165 21323 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.894687 21323 ts_tablet_manager.cc:1403] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.004s
I20251214 02:58:29.895620 21327 tablet_bootstrap.cc:654] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.897373 21323 raft_consensus.cc:359] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.898082 21323 raft_consensus.cc:385] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.898337 21323 raft_consensus.cc:740] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.898986 21323 consensus_queue.cc:260] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.901643 21323 ts_tablet_manager.cc:1434] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:58:29.902609 21323 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.902937 21322 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.903362 21322 ts_tablet_manager.cc:1403] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.011s
I20251214 02:58:29.905627 21322 raft_consensus.cc:359] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.906224 21322 raft_consensus.cc:385] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.906466 21322 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.907193 21322 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.908042 21323 tablet_bootstrap.cc:654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.909390 21322 ts_tablet_manager.cc:1434] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:29.910343 21322 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.910079 21028 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:29.911788 21327 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.912209 21327 ts_tablet_manager.cc:1403] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.002s
I20251214 02:58:29.917302 21348 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.917826 21348 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.919912 21348 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.922221 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:29.923482 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:29.924207 21327 raft_consensus.cc:359] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.924824 21327 raft_consensus.cc:385] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.922360 21322 tablet_bootstrap.cc:654] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.925168 21327 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.925985 21327 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.928074 21327 ts_tablet_manager.cc:1434] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20251214 02:58:29.928953 21327 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
W20251214 02:58:29.925122 21173 leader_election.cc:343] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:29.933548 21173 leader_election.cc:343] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.934024 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:29.935708 21328 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:29.934275 21323 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:29.936847 21323 ts_tablet_manager.cc:1403] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.000s
I20251214 02:58:29.939298 21323 raft_consensus.cc:359] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.939859 21327 tablet_bootstrap.cc:654] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.944829 21323 raft_consensus.cc:385] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.945190 21323 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.946070 21322 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.945961 21323 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.946543 21322 ts_tablet_manager.cc:1403] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.007s
I20251214 02:58:29.928886 21029 catalog_manager.cc:5654] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:29.948761 21323 ts_tablet_manager.cc:1434] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251214 02:58:29.948202 21029 catalog_manager.cc:5654] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:29.949069 21322 raft_consensus.cc:359] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.949704 21322 raft_consensus.cc:385] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.949818 21323 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:29.949975 21322 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.950778 21322 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.952842 21322 ts_tablet_manager.cc:1434] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20251214 02:58:29.953017 21327 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.953413 21327 ts_tablet_manager.cc:1403] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20251214 02:58:29.953692 21322 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.954205 21397 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.954667 21397 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.956247 21327 raft_consensus.cc:359] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.956588 21397 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:29.956882 21327 raft_consensus.cc:385] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.957171 21327 raft_consensus.cc:740] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.958381 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:29.959048 21205 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:29.959477 21322 tablet_bootstrap.cc:654] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.960635 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:29.962481 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:29.963933 21098 leader_election.cc:343] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.964859 21397 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:29.965258 21397 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:29.965626 21397 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.965807 21322 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:29.966228 21322 ts_tablet_manager.cc:1403] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20251214 02:58:29.966833 21327 consensus_queue.cc:260] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.968604 21322 raft_consensus.cc:359] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.969187 21327 ts_tablet_manager.cc:1434] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.016s user 0.001s sys 0.005s
I20251214 02:58:29.969220 21322 raft_consensus.cc:385] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.969691 21322 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:29.970331 21327 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:29.970386 21322 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.976727 21323 tablet_bootstrap.cc:654] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.980299 21322 ts_tablet_manager.cc:1434] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20251214 02:58:29.977530 21397 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.981248 21322 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:29.983500 21327 tablet_bootstrap.cc:654] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.988822 21322 tablet_bootstrap.cc:654] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:29.992173 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:29.993072 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:58:29.993646 21098 leader_election.cc:343] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.993707 21205 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:29.994627 21327 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:29.995057 21327 ts_tablet_manager.cc:1403] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.006s
I20251214 02:58:29.997140 21433 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:29.997470 21327 raft_consensus.cc:359] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.997562 21433 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:29.998128 21327 raft_consensus.cc:385] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:29.999517 21433 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.001554 21323 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:30.001358 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.001967 21323 ts_tablet_manager.cc:1403] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.011s
W20251214 02:58:30.003093 21248 leader_election.cc:343] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.004374 21323 raft_consensus.cc:359] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.004829 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.005019 21323 raft_consensus.cc:385] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.006047 21323 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
W20251214 02:58:30.006181 21247 leader_election.cc:343] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:29.998384 21327 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.006691 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:30.006810 21323 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.007401 21327 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.008070 21433 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.008947 21323 ts_tablet_manager.cc:1434] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251214 02:58:30.009469 21327 ts_tablet_manager.cc:1434] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20251214 02:58:30.009958 21323 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:30.010337 21327 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:30.012288 21205 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.017252 21397 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.017915 21322 tablet_bootstrap.cc:492] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.018057 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:30.018447 21322 ts_tablet_manager.cc:1403] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.006s
I20251214 02:58:30.016289 21323 tablet_bootstrap.cc:654] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.020864 21322 raft_consensus.cc:359] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.021526 21322 raft_consensus.cc:385] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.021777 21322 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.022744 21327 tablet_bootstrap.cc:654] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.022434 21322 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.024592 21322 ts_tablet_manager.cc:1434] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:30.022408 21397 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.027732 21322 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:30.028344 21397 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.029196 21397 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.036685 21322 tablet_bootstrap.cc:654] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.046181 21029 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.048435 21323 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:30.048892 21323 ts_tablet_manager.cc:1403] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.008s
I20251214 02:58:30.051739 21323 raft_consensus.cc:359] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.052381 21323 raft_consensus.cc:385] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.052655 21323 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.053454 21327 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:30.053431 21323 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.053994 21327 ts_tablet_manager.cc:1403] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.016s
I20251214 02:58:30.056432 21327 raft_consensus.cc:359] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.057044 21327 raft_consensus.cc:385] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.057288 21327 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.057919 21327 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.058549 21323 ts_tablet_manager.cc:1434] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20251214 02:58:30.059448 21323 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:30.060175 21327 ts_tablet_manager.cc:1434] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20251214 02:58:30.060400 21322 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.060727 21322 ts_tablet_manager.cc:1403] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20251214 02:58:30.061614 21327 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:30.062985 21322 raft_consensus.cc:359] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.063603 21322 raft_consensus.cc:385] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.063863 21322 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.064803 21323 tablet_bootstrap.cc:654] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.064800 21322 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.067664 21327 tablet_bootstrap.cc:654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.070878 21397 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.071339 21397 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.073208 21397 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.073940 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.074535 21205 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.074625 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.075783 21099 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.076035 21322 ts_tablet_manager.cc:1434] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
W20251214 02:58:30.076539 21098 leader_election.cc:343] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.076766 21397 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.077135 21397 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.077454 21397 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.084442 21397 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.085358 21322 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:30.086264 21397 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.087809 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.099483 21205 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.085664 21433 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.100589 21433 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.087358 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:30.103964 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:30.105319 21098 leader_election.cc:343] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.106189 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.106896 21204 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:30.107736 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:30.102497 21433 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.108438 21375 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.094492 21327 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:30.109486 21433 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.110343 21327 ts_tablet_manager.cc:1403] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.006s
I20251214 02:58:30.112838 21327 raft_consensus.cc:359] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.113451 21327 raft_consensus.cc:385] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.113705 21327 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.114406 21327 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.116582 21327 ts_tablet_manager.cc:1434] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:30.117547 21327 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:30.119884 21433 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.119899 21397 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.120529 21397 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.120507 21433 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.122505 21433 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.125871 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.126922 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.127108 21322 tablet_bootstrap.cc:654] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.127538 21130 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:30.128814 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.129390 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.129773 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.130091 21282 raft_consensus.cc:2393] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:30.130395 21433 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.130753 21433 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.131062 21433 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.131247 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:58:30.131569 21248 leader_election.cc:343] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.131990 21437 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.135120 21205 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:58:30.138203 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:30.139178 21099 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:30.139842 21375 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.126945 21397 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.140241 21360 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.140937 21360 raft_consensus.cc:697] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.141350 21397 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.141785 21360 consensus_queue.cc:237] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.147665 21327 tablet_bootstrap.cc:654] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.141820 21397 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.151839 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.152545 21282 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.151839 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.153846 21205 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.158115 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.158882 21347 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.159217 21347 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.159508 21347 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.160194 21397 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.167495 21347 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.168888 21432 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.138361 21437 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.173070 21322 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.173509 21322 ts_tablet_manager.cc:1403] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.002s
I20251214 02:58:30.175869 21322 raft_consensus.cc:359] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.176484 21322 raft_consensus.cc:385] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.176739 21322 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.177394 21322 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.179536 21322 ts_tablet_manager.cc:1434] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20251214 02:58:30.179787 21347 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.182227 21323 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:30.182757 21323 ts_tablet_manager.cc:1403] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.004s
I20251214 02:58:30.184206 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.184404 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:30.184808 21204 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.184901 21282 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.143608 21433 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.188125 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:30.188673 21130 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.191689 21282 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.193035 21322 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:30.193552 21098 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.195803 21436 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.196225 21436 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.196688 21347 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.198313 21436 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.198496 21347 raft_consensus.cc:697] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.199322 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.199779 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.199924 21280 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.200357 21205 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.201280 21098 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.202215 21436 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.202512 21322 tablet_bootstrap.cc:654] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.202602 21436 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.202342 21347 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.203015 21436 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.196254 21204 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.205870 21327 tablet_bootstrap.cc:492] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:30.206318 21327 ts_tablet_manager.cc:1403] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.014s
I20251214 02:58:30.208812 21327 raft_consensus.cc:359] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.209419 21327 raft_consensus.cc:385] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.209694 21327 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.210273 21327 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.212371 21327 ts_tablet_manager.cc:1434] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:30.212790 21348 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.213232 21348 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.212368 21436 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.215873 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.216464 21127 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:30.216955 21436 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.217916 21348 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.218806 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.219010 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:30.219805 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.219853 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.220356 21280 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.220466 21204 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.220539 21282 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:30.227646 21348 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.228087 21348 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.228451 21348 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.232971 21328 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.237141 21204 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.238020 21130 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:30.238209 21099 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.239280 21436 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.239290 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.239849 21436 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.240293 21432 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.240772 21432 raft_consensus.cc:697] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:30.240641 21436 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.241593 21432 consensus_queue.cc:237] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.243382 21322 tablet_bootstrap.cc:492] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.243829 21322 ts_tablet_manager.cc:1403] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.010s
I20251214 02:58:30.246894 21280 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.213239 21327 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:30.255497 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.247088 21322 raft_consensus.cc:359] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.257227 21322 raft_consensus.cc:385] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.257517 21322 raft_consensus.cc:740] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
W20251214 02:58:30.257884 21248 leader_election.cc:343] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.257591 21433 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.258258 21322 consensus_queue.cc:260] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.260739 21322 ts_tablet_manager.cc:1434] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251214 02:58:30.261601 21322 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:30.267017 21322 tablet_bootstrap.cc:654] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.271560 21327 tablet_bootstrap.cc:654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.253510 21323 raft_consensus.cc:359] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.254328 21349 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.273672 21323 raft_consensus.cc:385] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.276177 21323 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.277083 21323 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.279238 21323 ts_tablet_manager.cc:1434] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.096s user 0.004s sys 0.001s
I20251214 02:58:30.280133 21323 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:30.251788 21348 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.287851 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:30.288424 21127 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.291980 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:30.292584 21279 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.292865 21348 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.291594 21029 catalog_manager.cc:5654] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.302554 21347 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.305833 21347 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.306927 21433 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.307438 21433 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.311465 21323 tablet_bootstrap.cc:654] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.312624 21433 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.315899 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.311021 21348 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20251214 02:58:30.318682 21247 leader_election.cc:343] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.318205 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:30.326804 21248 leader_election.cc:343] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.327221 21248 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:30.328351 21433 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.330399 21434 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 02:58:30.351266 21436 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.351811 21436 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.353804 21436 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.356747 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:30.366268 21098 leader_election.cc:343] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.367906 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.368579 21204 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.369848 21099 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:30.370980 21347 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.371336 21347 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.371652 21347 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.339196 21279 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:30.386258 21347 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.389344 21347 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.392078 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:58:30.397293 21098 leader_election.cc:343] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.405335 21029 catalog_manager.cc:5654] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:30.338044 21127 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:30.409596 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.409751 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.410336 21205 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.417742 21348 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.418607 21348 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:30.419450 21348 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.443423 21347 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.445178 21322 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.445640 21322 ts_tablet_manager.cc:1403] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.184s user 0.018s sys 0.045s
I20251214 02:58:30.448305 21322 raft_consensus.cc:359] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.448997 21322 raft_consensus.cc:385] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.449257 21322 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.482414 21433 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.483064 21433 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.500850 21433 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.531391 21347 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.531987 21347 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.569154 21205 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.612600 21323 tablet_bootstrap.cc:492] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:30.613636 21323 ts_tablet_manager.cc:1403] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.334s user 0.056s sys 0.094s
I20251214 02:58:30.616808 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.617723 21205 raft_consensus.cc:2393] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:30.610553 21322 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.620126 21433 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.620611 21433 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.621673 21327 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:30.622133 21327 ts_tablet_manager.cc:1403] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.409s user 0.015s sys 0.014s
I20251214 02:58:30.624657 21327 raft_consensus.cc:359] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.625046 21433 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.625305 21327 raft_consensus.cc:385] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.625934 21327 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.626633 21327 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.627938 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.628571 21205 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:30.629027 21327 ts_tablet_manager.cc:1434] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:58:30.616979 21347 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.634409 21322 ts_tablet_manager.cc:1434] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.189s user 0.006s sys 0.001s
I20251214 02:58:30.635478 21322 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:30.642851 21099 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:30.643389 21323 raft_consensus.cc:359] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.644062 21323 raft_consensus.cc:385] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.644343 21323 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.644548 21439 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.645008 21439 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.644966 21323 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.645874 21439 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.649296 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:30.649904 21204 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:30.647244 21323 ts_tablet_manager.cc:1434] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20251214 02:58:30.654966 21099 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.655339 21323 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:30.655723 21437 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.656061 21437 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.656386 21437 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.656991 21435 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.657436 21435 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.658725 21360 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.659808 21435 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.659965 21360 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.676086 21437 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:30.680671 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 62168.8us
I20251214 02:58:30.684625 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.685242 21208 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.687894 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.688689 21130 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:30.695163 21208 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:30.696429 21099 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:30.697510 21347 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.698056 21347 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:30.698746 21323 tablet_bootstrap.cc:654] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.699225 21347 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.706482 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.640969 21027 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.709769 21027 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.629973 21327 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:30.724449 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.726415 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.730533 21437 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.736756 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:30.737708 21375 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:30.738087 21375 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:30.738412 21375 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.746505 21375 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.762102 21437 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 02:58:30.763759 21098 leader_election.cc:343] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.766857 21247 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:30.769882 21247 leader_election.cc:343] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.770996 21433 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.777925 21029 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.784147 21433 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.842123 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.874296 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:30.877012 21323 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:30.877547 21323 ts_tablet_manager.cc:1403] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.222s user 0.011s sys 0.032s
I20251214 02:58:30.801369 21375 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:30.885686 21439 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.888638 21323 raft_consensus.cc:359] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.889300 21323 raft_consensus.cc:385] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.889550 21323 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.890123 21323 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:30.892887 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49034.1us
W20251214 02:58:30.894057 21173 leader_election.cc:343] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.894472 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:30.895152 21434 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:30.897078 21327 tablet_bootstrap.cc:654] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.898464 21328 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:30.898936 21328 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.900760 21328 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:30.904521 21208 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.911243 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:30.911825 21207 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:30.918496 21207 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:30.923354 21323 ts_tablet_manager.cc:1434] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20251214 02:58:30.924399 21323 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:30.928279 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:30.935135 21322 tablet_bootstrap.cc:654] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:30.942678 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:30.943456 21127 raft_consensus.cc:2393] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:30.944998 21433 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 02:58:30.946949 21247 leader_election.cc:343] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.951125 21327 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:30.952313 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:30.953591 21433 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:30.954360 21433 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:30.955313 21433 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.951615 21327 ts_tablet_manager.cc:1403] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.322s user 0.012s sys 0.031s
I20251214 02:58:30.954893 21027 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.959779 21027 catalog_manager.cc:5654] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:30.961313 21323 tablet_bootstrap.cc:654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:30.966325 21098 leader_election.cc:343] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:30.966948 21439 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.967707 21436 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.968159 21439 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.970362 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:30.972858 21432 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:30.993566 21322 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:30.993274 21327 raft_consensus.cc:359] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:30.994112 21322 ts_tablet_manager.cc:1403] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.359s user 0.006s sys 0.004s
I20251214 02:58:30.994118 21327 raft_consensus.cc:385] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:30.995319 21327 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:30.996079 21327 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.027067 21322 raft_consensus.cc:359] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.027936 21322 raft_consensus.cc:385] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.028218 21322 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.031435 21322 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.035460 21322 ts_tablet_manager.cc:1434] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20251214 02:58:31.036468 21322 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.038955 21327 ts_tablet_manager.cc:1434] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
I20251214 02:58:31.039950 21327 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.044996 21327 tablet_bootstrap.cc:654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.051709 21322 tablet_bootstrap.cc:654] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.057624 21327 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.058084 21327 ts_tablet_manager.cc:1403] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.002s
I20251214 02:58:31.261391 21397 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.261894 21397 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.266355 21360 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:31.268157 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.269956 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.272527 21344 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 02:58:31.278770 21099 leader_election.cc:343] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.283295 21397 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.291714 21434 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:31.303894 21348 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.304456 21348 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.306456 21348 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.309906 21328 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:31.309840 21397 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:31.327400 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.329449 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.330103 21278 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
W20251214 02:58:31.342308 21173 leader_election.cc:343] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.360793 21173 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.363982 21434 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.364446 21434 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:31.364789 21434 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.370790 21322 tablet_bootstrap.cc:492] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.371291 21322 ts_tablet_manager.cc:1403] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.335s user 0.015s sys 0.000s
I20251214 02:58:31.373780 21322 raft_consensus.cc:359] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:31.363507 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 78629.1us
I20251214 02:58:31.374418 21322 raft_consensus.cc:385] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.389986 21322 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.400245 21327 raft_consensus.cc:359] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.400987 21327 raft_consensus.cc:385] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.401346 21327 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.401943 21327 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.390689 21322 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.410534 21323 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.411017 21322 ts_tablet_manager.cc:1434] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20251214 02:58:31.411940 21322 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.417795 21322 tablet_bootstrap.cc:654] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.418406 21327 ts_tablet_manager.cc:1434] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.360s user 0.009s sys 0.000s
I20251214 02:58:31.419344 21327 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.411038 21323 ts_tablet_manager.cc:1403] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.487s user 0.029s sys 0.002s
I20251214 02:58:31.430680 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.431362 21281 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:31.426170 21323 raft_consensus.cc:359] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.432533 21323 raft_consensus.cc:385] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.432837 21323 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.433322 21281 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.433432 21323 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:31.433523 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 72841.3us
I20251214 02:58:31.438529 21323 ts_tablet_manager.cc:1434] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20251214 02:58:31.439428 21323 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
W20251214 02:58:31.443027 21098 consensus_peers.cc:597] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 -> Peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Couldn't send request to peer 6e36caefa37d47a5968cffd9e270044f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:31.443747 21098 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:31.445217 21327 tablet_bootstrap.cc:654] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.446198 21437 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.446594 21437 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:31.446920 21437 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.448410 21347 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:31.453961 21360 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.454397 21360 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.455493 21323 tablet_bootstrap.cc:654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.456272 21360 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.457139 21322 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.457587 21322 ts_tablet_manager.cc:1403] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.008s
I20251214 02:58:31.459977 21322 raft_consensus.cc:359] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.460382 21327 tablet_bootstrap.cc:492] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.460642 21322 raft_consensus.cc:385] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.460996 21327 ts_tablet_manager.cc:1403] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20251214 02:58:31.461673 21434 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.462319 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.462939 21207 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:31.463830 21327 raft_consensus.cc:359] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.464049 21099 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:31.464639 21327 raft_consensus.cc:385] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.466332 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:31.475265 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:31.460958 21322 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.475772 21347 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.476118 21347 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:58:31.476576 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:31.476274 21322 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.468314 21434 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.471129 21323 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.475468 21327 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.467334 21437 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.477684 21327 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.478542 21128 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.478727 21322 ts_tablet_manager.cc:1434] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20251214 02:58:31.479652 21322 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.479974 21327 ts_tablet_manager.cc:1434] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.019s user 0.003s sys 0.002s
I20251214 02:58:31.480893 21327 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.483649 21347 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.485064 21322 tablet_bootstrap.cc:654] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.485562 21128 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:31.468356 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.486835 21279 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:31.486985 21432 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.487102 21323 ts_tablet_manager.cc:1403] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.009s
I20251214 02:58:31.487481 21432 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.487761 21278 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.486851 21327 tablet_bootstrap.cc:654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.488932 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:31.489362 21432 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:31.489565 21279 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.495754 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.496068 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.495754 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:31.496587 21173 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:31.497269 21248 leader_election.cc:343] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.497534 21434 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 02:58:31.497968 21247 leader_election.cc:343] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.498060 21434 raft_consensus.cc:697] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:31.498416 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.498975 21434 consensus_queue.cc:237] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.499214 21432 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.501492 21279 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:31.502488 21098 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:31.504763 21278 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:31.505496 21360 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:31.506047 21327 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.506348 21360 raft_consensus.cc:697] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:31.506515 21327 ts_tablet_manager.cc:1403] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.004s
I20251214 02:58:31.506985 21323 raft_consensus.cc:359] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.507675 21323 raft_consensus.cc:385] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.507367 21360 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.508183 21323 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.508823 21323 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.509724 21327 raft_consensus.cc:359] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.510372 21327 raft_consensus.cc:385] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.510655 21327 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.511927 21323 ts_tablet_manager.cc:1434] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.024s user 0.003s sys 0.002s
I20251214 02:58:31.512821 21323 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.512979 21322 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
W20251214 02:58:31.513208 21099 leader_election.cc:343] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.513458 21322 ts_tablet_manager.cc:1403] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.006s
I20251214 02:58:31.516310 21322 raft_consensus.cc:359] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.516948 21322 raft_consensus.cc:385] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.517225 21322 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.518303 21323 tablet_bootstrap.cc:654] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.518747 21327 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.520982 21327 ts_tablet_manager.cc:1434] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251214 02:58:31.521885 21327 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.517896 21322 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.531929 21322 ts_tablet_manager.cc:1434] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20251214 02:58:31.532969 21322 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.487742 21437 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.494277 21347 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.538954 21322 tablet_bootstrap.cc:654] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.540547 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:31.540486 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:31.541283 21205 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.543092 21327 tablet_bootstrap.cc:654] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:31.546277 21247 consensus_peers.cc:597] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f -> Peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Couldn't send request to peer 721d604130924d939d291fe3eef8f6d1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:31.548180 21205 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:31.549532 21099 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:31.550465 21437 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:31.551000 21437 raft_consensus.cc:697] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:31.551944 21322 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.552444 21322 ts_tablet_manager.cc:1403] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20251214 02:58:31.552166 21437 consensus_queue.cc:237] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.557228 21281 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.557665 21327 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.559548 21327 ts_tablet_manager.cc:1403] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20251214 02:58:31.560941 21347 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.561457 21432 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:31.562016 21327 raft_consensus.cc:359] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.562645 21327 raft_consensus.cc:385] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.562888 21327 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.564235 21322 raft_consensus.cc:359] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.564905 21322 raft_consensus.cc:385] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.565155 21322 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.565769 21322 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.566788 21327 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.569729 21322 ts_tablet_manager.cc:1434] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.017s user 0.003s sys 0.004s
I20251214 02:58:31.570679 21322 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.574844 21327 ts_tablet_manager.cc:1434] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20251214 02:58:31.575834 21327 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.581400 21375 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.581857 21375 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.583781 21375 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:31.585268 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.586330 21327 tablet_bootstrap.cc:654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:31.587970 21247 leader_election.cc:343] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.585516 21322 tablet_bootstrap.cc:654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.588784 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:31.590067 21248 leader_election.cc:343] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.590479 21248 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.591543 21375 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.597546 21327 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.597929 21327 ts_tablet_manager.cc:1403] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.005s
I20251214 02:58:31.599884 21327 raft_consensus.cc:359] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.600340 21327 raft_consensus.cc:385] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.600546 21327 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.601022 21327 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.602509 21327 ts_tablet_manager.cc:1434] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 02:58:31.603523 21327 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.609335 21327 tablet_bootstrap.cc:654] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.615729 21347 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.616290 21347 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.618377 21347 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.619182 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.619716 21347 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.620280 21347 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:31.620589 21099 leader_election.cc:343] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.623086 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.623360 21347 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.623375 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.624280 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:31.624850 21098 leader_election.cc:343] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.625299 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:58:31.625404 21099 leader_election.cc:343] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.624878 21279 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:31.627560 21098 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:31.628347 21360 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.628973 21360 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.629333 21360 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:31.629667 21360 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.646723 21360 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.665603 21360 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.666220 21360 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.666549 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:31.666755 21360 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:31.668886 21099 leader_election.cc:343] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.670387 21360 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.670243 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:31.671633 21099 leader_election.cc:343] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.673435 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:31.674072 21279 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.678826 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.607828 21281 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:58:31.680166 21098 leader_election.cc:343] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.680590 21098 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:31.682459 21437 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.682750 21279 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:31.683442 21323 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.683884 21323 ts_tablet_manager.cc:1403] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.171s user 0.100s sys 0.043s
I20251214 02:58:31.683738 21098 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:31.684541 21437 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:31.685066 21437 raft_consensus.cc:697] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:31.686550 21323 raft_consensus.cc:359] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.687038 21322 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.687248 21323 raft_consensus.cc:385] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.687551 21323 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.687671 21322 ts_tablet_manager.cc:1403] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.117s user 0.020s sys 0.035s
I20251214 02:58:31.687309 21437 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.685436 21029 catalog_manager.cc:5654] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:31.688027 21323 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.689419 21327 tablet_bootstrap.cc:492] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.689579 21323 ts_tablet_manager.cc:1434] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251214 02:58:31.689849 21327 ts_tablet_manager.cc:1403] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.087s user 0.031s sys 0.043s
I20251214 02:58:31.690313 21323 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.690457 21322 raft_consensus.cc:359] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.691629 21322 raft_consensus.cc:385] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.691882 21322 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.692521 21322 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.693001 21327 raft_consensus.cc:359] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.693625 21327 raft_consensus.cc:385] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.693898 21327 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.694943 21322 ts_tablet_manager.cc:1434] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251214 02:58:31.694659 21327 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.696077 21322 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.696678 21323 tablet_bootstrap.cc:654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.696693 21327 ts_tablet_manager.cc:1434] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:58:31.697499 21029 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:58:31.698650 21327 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.701813 21322 tablet_bootstrap.cc:654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.705709 21375 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.706207 21375 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.707409 21323 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.707840 21323 ts_tablet_manager.cc:1403] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20251214 02:58:31.708251 21375 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:31.709918 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.710283 21323 raft_consensus.cc:359] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.710547 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:31.711393 21247 leader_election.cc:343] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:31.711889 21248 leader_election.cc:343] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.704777 21327 tablet_bootstrap.cc:654] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.712316 21248 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.713321 21375 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.713999 21323 raft_consensus.cc:385] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.714283 21323 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.714910 21323 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.717036 21323 ts_tablet_manager.cc:1434] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251214 02:58:31.717893 21323 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.721009 21027 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:31.723317 21323 tablet_bootstrap.cc:654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.723482 21027 catalog_manager.cc:5654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:31.724987 21322 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.725486 21322 ts_tablet_manager.cc:1403] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.000s
I20251214 02:58:31.728375 21322 raft_consensus.cc:359] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.729028 21322 raft_consensus.cc:385] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.729287 21322 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.729873 21322 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.732144 21322 ts_tablet_manager.cc:1434] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251214 02:58:31.733973 21322 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.734452 21323 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.734905 21323 ts_tablet_manager.cc:1403] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20251214 02:58:31.735615 21327 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.736044 21327 ts_tablet_manager.cc:1403] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.005s
I20251214 02:58:31.737251 21323 raft_consensus.cc:359] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.737877 21323 raft_consensus.cc:385] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.738116 21323 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.738468 21327 raft_consensus.cc:359] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.739109 21327 raft_consensus.cc:385] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.738888 21323 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.739427 21322 tablet_bootstrap.cc:654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.739387 21327 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.740411 21327 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.740978 21323 ts_tablet_manager.cc:1434] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20251214 02:58:31.742003 21323 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.742774 21327 ts_tablet_manager.cc:1434] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251214 02:58:31.743757 21327 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.747479 21323 tablet_bootstrap.cc:654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.750784 21327 tablet_bootstrap.cc:654] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.752425 21322 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.752861 21322 ts_tablet_manager.cc:1403] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20251214 02:58:31.755710 21322 raft_consensus.cc:359] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.756314 21322 raft_consensus.cc:385] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.756309 21327 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.756697 21322 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.757059 21327 ts_tablet_manager.cc:1403] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20251214 02:58:31.757355 21322 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.758415 21323 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.758846 21323 ts_tablet_manager.cc:1403] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20251214 02:58:31.759733 21322 ts_tablet_manager.cc:1434] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251214 02:58:31.759986 21327 raft_consensus.cc:359] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.760700 21327 raft_consensus.cc:385] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.760991 21327 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.761042 21322 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.761103 21323 raft_consensus.cc:359] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.761857 21323 raft_consensus.cc:385] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.762006 21327 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.764299 21327 ts_tablet_manager.cc:1434] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 02:58:31.765200 21327 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.765159 21279 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.767092 21322 tablet_bootstrap.cc:654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.771906 21327 tablet_bootstrap.cc:654] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.762145 21323 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.774040 21323 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.776723 21323 ts_tablet_manager.cc:1434] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20251214 02:58:31.780623 21323 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.802655 21322 tablet_bootstrap.cc:492] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.802935 21323 tablet_bootstrap.cc:654] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.803125 21322 ts_tablet_manager.cc:1403] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.019s
I20251214 02:58:31.805661 21322 raft_consensus.cc:359] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.809302 21322 raft_consensus.cc:385] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.809587 21322 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.810293 21322 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.812638 21322 ts_tablet_manager.cc:1434] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251214 02:58:31.813517 21322 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.818934 21322 tablet_bootstrap.cc:654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.833230 21322 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.833676 21322 ts_tablet_manager.cc:1403] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20251214 02:58:31.836840 21327 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.837365 21327 ts_tablet_manager.cc:1403] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.072s user 0.022s sys 0.045s
I20251214 02:58:31.837604 21322 raft_consensus.cc:359] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.838205 21323 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.838272 21322 raft_consensus.cc:385] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.838726 21323 ts_tablet_manager.cc:1403] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.005s
I20251214 02:58:31.838685 21322 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.839969 21327 raft_consensus.cc:359] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.840761 21327 raft_consensus.cc:385] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.841053 21327 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.841460 21323 raft_consensus.cc:359] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.842072 21323 raft_consensus.cc:385] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.842397 21323 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.842123 21327 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.842978 21323 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.843822 21322 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.844451 21375 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.844617 21323 ts_tablet_manager.cc:1434] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:31.845609 21323 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.845893 21322 ts_tablet_manager.cc:1434] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251214 02:58:31.846787 21322 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.851464 21323 tablet_bootstrap.cc:654] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.851598 21375 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.852222 21322 tablet_bootstrap.cc:654] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.853727 21375 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:31.855109 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.855706 21327 ts_tablet_manager.cc:1434] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
W20251214 02:58:31.856375 21248 leader_election.cc:343] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.856884 21327 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.859735 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.864557 21327 tablet_bootstrap.cc:654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:31.864609 21247 leader_election.cc:343] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:31.865177 21247 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.867039 21375 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.874459 21322 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.874955 21322 ts_tablet_manager.cc:1403] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.000s
I20251214 02:58:31.875047 21323 tablet_bootstrap.cc:492] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:31.875597 21323 ts_tablet_manager.cc:1403] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.009s
I20251214 02:58:31.870894 21027 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:31.877723 21322 raft_consensus.cc:359] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.878376 21322 raft_consensus.cc:385] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.878057 21323 raft_consensus.cc:359] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.878667 21322 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.878793 21323 raft_consensus.cc:385] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.879122 21323 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.879341 21322 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.881059 21434 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.881188 21323 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.881716 21434 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.881911 21322 ts_tablet_manager.cc:1434] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251214 02:58:31.883257 21322 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:31.883767 21434 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.883803 21323 ts_tablet_manager.cc:1434] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251214 02:58:31.884981 21323 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:31.885572 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.885928 21327 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:31.886474 21327 ts_tablet_manager.cc:1403] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.005s
I20251214 02:58:31.886977 21434 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.887449 21434 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.889183 21434 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.889050 21327 raft_consensus.cc:359] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.889837 21327 raft_consensus.cc:385] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.890163 21327 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.890884 21327 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.891500 21322 tablet_bootstrap.cc:654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.892515 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.893106 21132 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:31.893276 21323 tablet_bootstrap.cc:654] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.894491 21327 ts_tablet_manager.cc:1434] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20251214 02:58:31.895524 21327 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:31.898761 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.899407 21278 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:31.887001 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:31.900895 21279 raft_consensus.cc:2393] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:31.902226 21327 tablet_bootstrap.cc:654] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:31.902618 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:31.904198 21434 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.904577 21434 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:31.904911 21434 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.910811 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:31.911628 21344 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.912199 21375 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.912660 21375 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.914508 21375 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:31.916019 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:31.916676 21128 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:31.917351 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.917693 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:31.917968 21205 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:31.918535 21375 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:31.918892 21375 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:31.919222 21375 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.921231 21322 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:31.921725 21344 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:31.921674 21322 ts_tablet_manager.cc:1403] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.004s
I20251214 02:58:31.925580 21434 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.927529 21434 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.928439 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:31.928982 21132 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.932250 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:31.932801 21278 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:31.934784 21439 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.935246 21439 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.936775 21132 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:31.937062 21439 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:31.931217 21322 raft_consensus.cc:359] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:31.938979 21322 raft_consensus.cc:385] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:31.939252 21322 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:31.943764 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.077312 21280 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:32.043895 21508 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:31.954699 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.078454 21508 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.079281 21323 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:32.079571 21278 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
W20251214 02:58:32.060299 21303 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4160 after lost signal to thread 19984
I20251214 02:58:32.080698 21508 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.077989 21327 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
W20251214 02:58:32.081737 21303 debug-util.cc:398] Leaking SignalData structure 0x7b0800041a80 after lost signal to thread 21065
I20251214 02:58:32.025163 21432 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:32.082891 21303 debug-util.cc:398] Leaking SignalData structure 0x7b0800155580 after lost signal to thread 21157
I20251214 02:58:31.967270 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:32.083976 21303 debug-util.cc:398] Leaking SignalData structure 0x7b080011dfc0 after lost signal to thread 21232
I20251214 02:58:32.084311 21344 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:32.084837 21344 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:58:32.084998 21303 debug-util.cc:398] Leaking SignalData structure 0x7b08001e3040 after lost signal to thread 21306
I20251214 02:58:32.085696 21344 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.079720 21323 ts_tablet_manager.cc:1403] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.195s user 0.017s sys 0.023s
I20251214 02:58:32.082665 21432 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.082018 21327 ts_tablet_manager.cc:1403] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.187s user 0.027s sys 0.007s
I20251214 02:58:32.090339 21323 raft_consensus.cc:359] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.090988 21323 raft_consensus.cc:385] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.090910 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.091490 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:31.945785 21437 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.146s
I20251214 02:58:32.092217 21208 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:32.092356 21327 raft_consensus.cc:359] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.093000 21099 leader_election.cc:343] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.093451 21248 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:32.094216 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.094856 21128 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:32.097209 21098 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:32.097952 21347 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.098328 21347 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:32.098681 21347 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.099256 21434 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.099711 21434 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.101495 21434 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.102454 21322 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.091364 21323 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.106422 21323 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.107937 21560 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:32.082077 21375 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.110944 21375 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.112684 21375 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.093005 21327 raft_consensus.cc:385] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.127975 21327 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.106120 21347 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.121033 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.127676 21323 ts_tablet_manager.cc:1434] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20251214 02:58:32.130793 21323 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
W20251214 02:58:32.119765 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 152617us
I20251214 02:58:32.128973 21327 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.138622 21434 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:32.142215 21375 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.163266 21375 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.163964 21560 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:32.146613 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 50295.3us
I20251214 02:58:32.165253 21205 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.165499 21347 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.158752 21434 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251214 02:58:32.130060 21439 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
W20251214 02:58:32.167990 21247 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.122774 21432 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.130321 21508 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.164481 21560 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.146802 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 53450.4us
I20251214 02:58:32.178797 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.179296 21127 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.179476 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.180537 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:32.181560 21439 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.182782 21029 catalog_manager.cc:5654] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:32.182389 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.186942 21132 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:32.177155 21508 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:32.190557 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.191591 21323 tablet_bootstrap.cc:654] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.192149 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.178797 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:32.192076 21279 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.193428 21322 ts_tablet_manager.cc:1434] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.271s user 0.052s sys 0.022s
I20251214 02:58:32.207625 21322 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:32.204248 21508 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.207679 21205 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.195183 21560 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.210259 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.196671 21439 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.211586 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.212821 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.229056 21282 raft_consensus.cc:2393] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:32.229872 21439 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:32.213382 21248 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.196967 21347 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.230600 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:32.230836 21347 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.232513 21248 leader_election.cc:343] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.233260 21432 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.234066 21347 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.234894 21508 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.236974 21508 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.208021 21128 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.242118 21327 ts_tablet_manager.cc:1434] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.152s user 0.016s sys 0.020s
I20251214 02:58:32.243155 21327 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:32.242432 21247 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:32.258108 21375 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.264580 21375 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.265969 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.266608 21208 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.276101 21508 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.276664 21508 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.266417 21375 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.322223 21574 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.329289 21574 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.314750 21565 raft_consensus.cc:2804] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.379258 21565 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:32.379655 21565 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.405690 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.406430 21278 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:32.401187 21434 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.407747 21434 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.417305 21344 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.417894 21344 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.382231 21349 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:32.385366 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 59591.8us
I20251214 02:58:32.496589 21508 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.526996 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.505003 21173 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:32.533895 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.487372 21349 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.537876 21349 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.539079 21349 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 02:58:32.542672 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 303845us
I20251214 02:58:32.544216 21128 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.547528 21322 tablet_bootstrap.cc:654] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.555228 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:32.559667 21248 leader_election.cc:343] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.502898 21574 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.577849 21327 tablet_bootstrap.cc:654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.556998 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:32.576174 21565 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.579859 21434 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:32.580576 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 384957us
I20251214 02:58:32.581791 21344 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:32.584018 21248 leader_election.cc:343] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.586795 21439 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.582450 21279 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:32.613473 21568 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 02:58:32.605118 21439 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:32.628149 21278 raft_consensus.cc:2393] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:32.624584 21347 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.628351 21439 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:32.625012 21099 leader_election.cc:343] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.646044 21099 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:32.625597 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:32.644855 21323 tablet_bootstrap.cc:492] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:32.651196 21323 ts_tablet_manager.cc:1403] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.521s user 0.034s sys 0.037s
I20251214 02:58:32.626730 21208 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.653928 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:32.654672 21360 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:32.659386 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.661564 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.670792 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.666887 21432 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
W20251214 02:58:32.674479 21099 leader_election.cc:343] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.649920 21347 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.638969 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 106578us
I20251214 02:58:32.666988 21128 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.687317 21565 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:58:32.664304 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 51458us
I20251214 02:58:32.691331 21347 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.677592 21432 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:32.708344 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.709054 21130 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:32.716080 21432 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.718196 21323 raft_consensus.cc:359] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.720551 21323 raft_consensus.cc:385] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.720790 21323 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.721256 21323 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.728519 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.732262 21322 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:32.732790 21322 ts_tablet_manager.cc:1403] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.525s user 0.022s sys 0.028s
W20251214 02:58:32.737053 21578 log.cc:927] Time spent T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20251214 02:58:32.737840 21205 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.735814 21439 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.742219 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:32.751581 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:32.752303 21131 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.756814 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:32.757668 21375 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.758013 21375 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:58:32.758070 21247 leader_election.cc:343] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.758395 21375 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.715579 21248 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
W20251214 02:58:32.761202 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 53453.7us
I20251214 02:58:32.766718 21323 ts_tablet_manager.cc:1434] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.115s user 0.004s sys 0.003s
W20251214 02:58:32.728080 21099 leader_election.cc:343] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.764232 21562 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:32.764546 21247 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:32.766687 21439 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:32.770960 21131 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.772924 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.765908 21128 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.773810 21375 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.787600 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.778319 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.788117 21375 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:32.788301 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.788940 21278 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:32.780755 21323 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:32.789160 21098 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:58:32.756812 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:32.795280 21572 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.794062 21562 raft_consensus.cc:697] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:32.795356 21583 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.796870 21562 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.800486 21327 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:32.800928 21327 ts_tablet_manager.cc:1403] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.558s user 0.019s sys 0.016s
I20251214 02:58:32.805543 21572 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:32.806051 21572 raft_consensus.cc:697] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:32.806871 21572 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.807577 21327 raft_consensus.cc:359] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.808246 21327 raft_consensus.cc:385] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.808498 21327 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.809051 21327 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.782483 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 57168us
I20251214 02:58:32.810864 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:32.819832 21173 leader_election.cc:343] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.819797 21322 raft_consensus.cc:359] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.820538 21322 raft_consensus.cc:385] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.820827 21322 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.821435 21322 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.828276 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.828949 21279 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:32.830652 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:32.830909 21247 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:32.831449 21127 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.831857 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.834544 21202 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.837323 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:32.840466 21327 ts_tablet_manager.cc:1434] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.039s user 0.004s sys 0.003s
I20251214 02:58:32.841470 21327 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:32.843205 21322 ts_tablet_manager.cc:1434] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.110s user 0.002s sys 0.005s
W20251214 02:58:32.820751 21173 leader_election.cc:343] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.844158 21322 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:32.849639 21028 catalog_manager.cc:5654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:32.833868 21432 raft_consensus.cc:2804] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:32.852167 21432 raft_consensus.cc:697] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:32.856266 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.865756 21375 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:32.862660 21432 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.870680 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.859820 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.861372 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:32.873296 21029 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:32.879367 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.879400 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:32.882421 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:32.885691 21098 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:32.881342 21276 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:32.885859 21173 leader_election.cc:343] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:32.875331 21248 leader_election.cc:343] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.891921 21437 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.874959 21375 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.897217 21375 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:58:32.897924 21099 leader_election.cc:343] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:32.898653 21099 leader_election.cc:343] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.898751 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:32.899096 21439 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:32.905843 21323 tablet_bootstrap.cc:654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.913484 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:32.913946 21127 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:32.914470 21347 raft_consensus.cc:2764] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20251214 02:58:32.918138 21590 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20251214 02:58:32.922835 21098 leader_election.cc:343] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.923358 21098 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:58:32.924690 21247 leader_election.cc:343] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.925053 21347 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.922181 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:32.929198 21375 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
W20251214 02:58:32.929312 21248 leader_election.cc:343] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.929733 21375 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:32.929729 21248 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:32.930585 21562 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.930720 21375 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.939179 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:32.941074 21173 leader_election.cc:343] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.944630 21173 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:32.945487 21173 leader_election.cc:343] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:32.945792 21173 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:32.948191 21344 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.950168 21580 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:32.950488 21276 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:32.951159 21599 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:32.951552 21599 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:32.952404 21599 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:32.941501 21349 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251214 02:58:32.957273 21349 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 02:58:32.959141 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:32.959897 21572 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:32.960358 21572 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:32.960896 21322 tablet_bootstrap.cc:654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.961158 21572 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:32.967629 21594 log.cc:927] Time spent T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20251214 02:58:32.977587 21327 tablet_bootstrap.cc:654] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:32.987990 21323 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:32.988499 21323 ts_tablet_manager.cc:1403] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.221s user 0.008s sys 0.008s
I20251214 02:58:32.997409 21323 raft_consensus.cc:359] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:32.998067 21323 raft_consensus.cc:385] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:32.998335 21323 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:32.998956 21323 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.007403 21323 ts_tablet_manager.cc:1434] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.019s user 0.000s sys 0.006s
I20251214 02:58:33.008459 21323 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:33.004535 21599 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.037864 21599 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.038887 21322 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:33.039427 21322 ts_tablet_manager.cc:1403] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.195s user 0.017s sys 0.004s
I20251214 02:58:33.040009 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:33.040627 21278 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.038800 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:58:33.043089 21173 leader_election.cc:343] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.046396 21327 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:33.046880 21327 ts_tablet_manager.cc:1403] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.206s user 0.005s sys 0.015s
I20251214 02:58:33.052630 21327 raft_consensus.cc:359] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.052872 21580 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.053644 21327 raft_consensus.cc:385] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.053804 21580 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.054718 21327 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.055644 21327 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.056308 21580 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.056986 21027 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:33.057826 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:33.058344 21327 ts_tablet_manager.cc:1434] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251214 02:58:33.058610 21127 raft_consensus.cc:2393] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:33.059358 21327 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:33.059409 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.059681 21278 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:33.062029 21323 tablet_bootstrap.cc:654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.066361 21322 raft_consensus.cc:359] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.067057 21322 raft_consensus.cc:385] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.067318 21322 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.067899 21322 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.070209 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:33.070425 21322 ts_tablet_manager.cc:1434] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20251214 02:58:33.071435 21322 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:33.071386 21599 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:33.072167 21599 raft_consensus.cc:697] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:33.071540 21028 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:33.072989 21599 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.073468 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:33.075371 21571 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.083356 21571 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:33.073347 21028 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:33.093189 21327 tablet_bootstrap.cc:654] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.093852 21322 tablet_bootstrap.cc:654] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.087350 21028 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:33.096858 21323 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:33.097342 21323 ts_tablet_manager.cc:1403] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.003s
I20251214 02:58:33.101846 21323 raft_consensus.cc:359] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.102491 21323 raft_consensus.cc:385] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.102774 21323 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.103356 21323 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.105998 21323 ts_tablet_manager.cc:1434] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20251214 02:58:33.106933 21327 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:33.107316 21327 ts_tablet_manager.cc:1403] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.005s
I20251214 02:58:33.107849 21323 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:33.111348 21327 raft_consensus.cc:359] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.111922 21327 raft_consensus.cc:385] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.112180 21327 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.112825 21327 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.115890 21322 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:33.117686 21322 ts_tablet_manager.cc:1403] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.002s
I20251214 02:58:33.121884 21327 ts_tablet_manager.cc:1434] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20251214 02:58:33.122632 21327 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:33.124854 21322 raft_consensus.cc:359] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.128543 21322 raft_consensus.cc:385] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.128965 21322 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.129676 21322 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.131881 21322 ts_tablet_manager.cc:1434] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20251214 02:58:33.132709 21322 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:33.135911 21572 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.136382 21572 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.143015 21572 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.146449 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.150523 21432 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.151021 21432 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.152863 21432 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:33.154767 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:33.156196 21247 leader_election.cc:343] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.156951 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:33.158288 21099 leader_election.cc:343] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:33.161322 21098 leader_election.cc:343] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.161893 21098 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:33.154748 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.168955 21437 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 02:58:33.170840 21248 leader_election.cc:343] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.171288 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:33.172336 21375 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:33.165963 21028 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:33.181404 21327 tablet_bootstrap.cc:654] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.181574 21323 tablet_bootstrap.cc:654] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.184793 21322 tablet_bootstrap.cc:654] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.218658 21323 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:33.219204 21323 ts_tablet_manager.cc:1403] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.112s user 0.018s sys 0.020s
I20251214 02:58:33.219461 21327 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:33.219897 21327 ts_tablet_manager.cc:1403] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.097s user 0.002s sys 0.006s
I20251214 02:58:33.222062 21347 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.222321 21327 raft_consensus.cc:359] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.222497 21347 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.223035 21327 raft_consensus.cc:385] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.223323 21327 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.224437 21347 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.225524 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.226234 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.226821 21202 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:33.224167 21327 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:33.226833 21098 leader_election.cc:343] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.228796 21327 ts_tablet_manager.cc:1434] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20251214 02:58:33.228791 21099 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:33.229826 21572 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:33.230250 21572 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:33.229918 21323 raft_consensus.cc:359] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.230649 21572 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.231038 21323 raft_consensus.cc:385] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.231683 21323 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.232342 21323 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.238410 21572 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.241061 21322 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:33.241698 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:33.244477 21205 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.247326 21322 ts_tablet_manager.cc:1403] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.115s user 0.015s sys 0.017s
I20251214 02:58:33.252986 21322 raft_consensus.cc:359] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.253602 21322 raft_consensus.cc:385] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.253856 21322 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.259923 21322 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.237884 21327 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:33.244814 21572 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.245741 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:33.265259 21347 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.248654 21323 ts_tablet_manager.cc:1434] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s
I20251214 02:58:33.247316 21202 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:33.270140 21098 leader_election.cc:343] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.272395 21202 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:33.273471 21099 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:33.273802 21327 tablet_bootstrap.cc:654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.274920 21347 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:33.275439 21347 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:33.276635 21322 ts_tablet_manager.cc:1434] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20251214 02:58:33.276367 21347 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.277539 21322 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:33.274977 21323 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:33.297523 21322 tablet_bootstrap.cc:654] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.300053 21347 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.310355 21323 tablet_bootstrap.cc:654] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.318627 21027 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:33.319447 21432 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.326908 21347 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.327353 21347 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.330343 21583 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.330862 21347 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.326500 21432 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 02:58:33.333119 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.333897 21278 raft_consensus.cc:2393] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:33.334175 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.338198 21098 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:33.343493 21437 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.378463 21322 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:33.379031 21322 ts_tablet_manager.cc:1403] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.021s
I20251214 02:58:33.383095 21203 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.387678 21437 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:33.389662 21432 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.397639 21327 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:33.398139 21327 ts_tablet_manager.cc:1403] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.160s user 0.018s sys 0.009s
I20251214 02:58:33.381985 21322 raft_consensus.cc:359] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.399482 21323 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:33.402933 21323 ts_tablet_manager.cc:1403] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.008s
I20251214 02:58:33.402261 21322 raft_consensus.cc:385] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.403846 21322 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.408262 21327 raft_consensus.cc:359] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.408955 21327 raft_consensus.cc:385] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.409211 21327 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.409773 21327 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.415149 21322 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.419152 21572 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.419528 21437 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.420006 21437 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.414494 21323 raft_consensus.cc:359] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.420884 21323 raft_consensus.cc:385] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.421177 21323 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.421748 21323 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.422173 21327 ts_tablet_manager.cc:1434] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.024s user 0.003s sys 0.003s
I20251214 02:58:33.423095 21327 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:33.423640 21437 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.424073 21323 ts_tablet_manager.cc:1434] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20251214 02:58:33.424332 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.425719 21323 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:33.426162 21279 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:33.427326 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.427460 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:33.428586 21099 leader_election.cc:343] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.429093 21437 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:33.429564 21437 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:33.430277 21437 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.444393 21375 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:33.451314 21322 ts_tablet_manager.cc:1434] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.072s user 0.004s sys 0.004s
I20251214 02:58:33.452109 21322 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:33.460057 21583 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.473546 21439 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.473899 21437 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.481245 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:33.481707 21322 tablet_bootstrap.cc:654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
W20251214 02:58:33.482379 21099 leader_election.cc:343] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.483829 21437 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.487704 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:33.488268 21278 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:33.493674 21580 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.498013 21432 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:33.502283 21580 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:33.506827 21432 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20251214 02:58:33.536646 21327 tablet_bootstrap.cc:654] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.543620 21323 tablet_bootstrap.cc:654] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.546018 21322 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:33.546484 21322 ts_tablet_manager.cc:1403] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.005s
I20251214 02:58:33.543992 21580 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.548208 21580 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.560384 21375 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.560863 21375 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.561995 21580 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:33.559055 21322 raft_consensus.cc:359] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.571343 21322 raft_consensus.cc:385] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.571663 21322 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.583921 21322 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.613008 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:33.599229 21278 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:33.613682 21130 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
W20251214 02:58:33.633000 21631 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20251214 02:58:33.638262 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:33.640394 21322 ts_tablet_manager.cc:1434] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.094s user 0.005s sys 0.023s
I20251214 02:58:33.641539 21322 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:33.635807 21583 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.650960 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:33.636644 21375 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:33.652678 21130 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:33.641331 21437 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:33.651777 21583 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.740285 21322 tablet_bootstrap.cc:654] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:33.699421 21173 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:33.760576 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 61272.8us
I20251214 02:58:33.748059 21375 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.705971 21437 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 02:58:33.764523 21612 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.114s user 0.003s sys 0.006s
I20251214 02:58:33.709991 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.768090 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:33.772689 21280 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:33.710404 21247 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:33.774181 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 108344us
I20251214 02:58:33.774299 21639 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:33.774821 21639 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:33.775174 21639 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:33.779395 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 102113us
I20251214 02:58:33.776997 21432 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.775665 21437 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.768431 21375 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.790612 21327 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:33.792596 21327 ts_tablet_manager.cc:1403] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.370s user 0.018s sys 0.033s
I20251214 02:58:33.793370 21432 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.797226 21432 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:33.798787 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:33.799474 21130 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:33.779978 21583 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:33.782676 21580 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:33.808038 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:33.819231 21248 leader_election.cc:343] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:33.806646 21580 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:33.827124 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.829907 21375 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:33.823422 21580 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:33.869442 21638 log.cc:927] Time spent T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
W20251214 02:58:33.875900 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 110322us
I20251214 02:58:33.859525 21583 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.948484 21327 raft_consensus.cc:359] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.965322 21327 raft_consensus.cc:385] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:33.965565 21327 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:33.966053 21327 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.944423 21439 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:33.966953 21439 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:33.968858 21439 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:33.948105 21628 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.056s user 0.003s sys 0.010s
I20251214 02:58:33.978320 21327 ts_tablet_manager.cc:1434] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.185s user 0.011s sys 0.009s
I20251214 02:58:33.979368 21327 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:33.981030 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:33.932147 21583 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:33.980029 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 88552.1us
I20251214 02:58:34.016353 21580 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.017588 21202 raft_consensus.cc:2393] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:34.019599 21583 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.019915 21323 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.020370 21323 ts_tablet_manager.cc:1403] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.595s user 0.017s sys 0.025s
W20251214 02:58:34.021303 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 133491us
I20251214 02:58:33.991488 21639 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.023635 21639 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.027102 21247 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:34.028861 21248 leader_election.cc:343] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.029247 21583 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.029589 21583 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.029914 21583 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:34.029783 21248 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.040277 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.040916 21127 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.043704 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.043898 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:58:34.047117 21248 leader_election.cc:343] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.018375 21580 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:34.047704 21248 leader_election.cc:343] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.047703 21323 raft_consensus.cc:359] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.049764 21323 raft_consensus.cc:385] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.050027 21323 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.050554 21323 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.065140 21323 ts_tablet_manager.cc:1434] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20251214 02:58:34.057376 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.066146 21323 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.067227 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.067229 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.067920 21282 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.072184 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.075551 21133 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:34.083024 21027 catalog_manager.cc:5654] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:34.089658 21127 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:34.091634 21322 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.092116 21322 ts_tablet_manager.cc:1403] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.451s user 0.029s sys 0.050s
I20251214 02:58:34.095293 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.097370 21327 tablet_bootstrap.cc:654] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.099016 21173 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.100080 21644 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.100520 21644 raft_consensus.cc:697] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:34.101296 21644 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:34.120568 21099 leader_election.cc:343] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:34.124953 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:34.131541 21583 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.132779 21279 raft_consensus.cc:2393] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:34.139983 21322 raft_consensus.cc:359] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.140972 21322 raft_consensus.cc:385] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.141160 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.141310 21322 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.142140 21322 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.144622 21323 tablet_bootstrap.cc:654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.144503 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.145026 21322 ts_tablet_manager.cc:1434] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.053s user 0.006s sys 0.001s
I20251214 02:58:34.145306 21132 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.146174 21322 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.152081 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.152501 21583 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.152969 21437 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:34.156998 21327 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.157505 21327 ts_tablet_manager.cc:1403] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.178s user 0.004s sys 0.007s
I20251214 02:58:34.157594 21322 tablet_bootstrap.cc:654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.167523 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.168419 21131 raft_consensus.cc:2393] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:58:34.171734 21247 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.172199 21247 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.172974 21583 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.176707 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:58:34.180023 21247 leader_election.cc:343] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.180439 21247 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.180984 21583 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.181109 21639 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.181423 21583 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.182828 21583 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.182750 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.183398 21132 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:34.183570 21437 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.183964 21437 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.184288 21437 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:34.189386 21098 consensus_peers.cc:597] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 -> Peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Couldn't send request to peer 6e36caefa37d47a5968cffd9e270044f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:34.191447 21583 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.191491 21247 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.193365 21583 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.193902 21639 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.191397 21437 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.194341 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:58:34.195658 21248 leader_election.cc:343] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.195830 21247 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.196408 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.196599 21639 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.197086 21639 raft_consensus.cc:697] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:58:34.198021 21099 leader_election.cc:343] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.197942 21639 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.198033 21437 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.200057 21327 raft_consensus.cc:359] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.200529 21327 raft_consensus.cc:385] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.200696 21327 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.201100 21327 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.202517 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.203380 21277 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:34.204668 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:34.209328 21649 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.209741 21323 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.210176 21323 ts_tablet_manager.cc:1403] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.144s user 0.003s sys 0.009s
I20251214 02:58:34.210879 21327 ts_tablet_manager.cc:1434] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.052s user 0.003s sys 0.003s
I20251214 02:58:34.213850 21327 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:34.216817 21323 raft_consensus.cc:359] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.218211 21323 raft_consensus.cc:385] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.218482 21323 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.221271 21323 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.223335 21323 ts_tablet_manager.cc:1434] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.013s user 0.005s sys 0.002s
I20251214 02:58:34.232174 21323 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.232818 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.233719 21132 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:34.235206 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.239526 21583 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.251672 21327 tablet_bootstrap.cc:654] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.256505 21323 tablet_bootstrap.cc:654] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.249480 21322 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.262253 21322 ts_tablet_manager.cc:1403] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.004s
I20251214 02:58:34.264236 21029 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:34.264643 21322 raft_consensus.cc:359] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.266991 21322 raft_consensus.cc:385] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.267261 21322 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.267860 21322 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.269953 21322 ts_tablet_manager.cc:1434] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251214 02:58:34.270835 21322 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.275745 21327 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.276211 21327 ts_tablet_manager.cc:1403] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.005s
I20251214 02:58:34.278486 21327 raft_consensus.cc:359] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.279206 21327 raft_consensus.cc:385] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.278645 21027 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:34.279490 21327 raft_consensus.cc:740] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.280078 21327 consensus_queue.cc:260] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.282123 21327 ts_tablet_manager.cc:1434] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251214 02:58:34.283510 21327 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:34.284785 21323 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.285215 21323 ts_tablet_manager.cc:1403] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.005s
I20251214 02:58:34.287606 21323 raft_consensus.cc:359] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.288242 21323 raft_consensus.cc:385] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.288511 21323 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.288710 21327 tablet_bootstrap.cc:654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.289214 21323 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.275879 21322 tablet_bootstrap.cc:654] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.293097 21323 ts_tablet_manager.cc:1434] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20251214 02:58:34.296567 21323 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.302444 21323 tablet_bootstrap.cc:654] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.304461 21644 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.314872 21644 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:34.305446 21099 consensus_peers.cc:597] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 -> Peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Couldn't send request to peer 668ad4f30bd649c88a034bbab9f13b20. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:34.306275 21322 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.317046 21322 ts_tablet_manager.cc:1403] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.003s
I20251214 02:58:34.318463 21644 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.319685 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.321619 21322 raft_consensus.cc:359] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.322350 21322 raft_consensus.cc:385] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.322683 21322 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.323568 21322 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.325402 21322 ts_tablet_manager.cc:1434] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20251214 02:58:34.326299 21322 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.327207 21437 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.327652 21437 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.320298 21279 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.330256 21173 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:34.333190 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.333787 21127 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.333873 21644 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.334240 21644 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.334556 21644 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.336042 21437 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.336359 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.338286 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.338908 21277 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.339957 21639 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.340138 21098 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.340443 21639 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.341904 21437 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.342242 21437 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.342538 21437 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.344470 21639 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.346225 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:34.349179 21248 leader_election.cc:343] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.349900 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:34.351867 21247 leader_election.cc:343] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.352293 21247 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.353452 21639 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.353685 21322 tablet_bootstrap.cc:654] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.361152 21639 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.361629 21639 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.363540 21322 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.363669 21327 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.364172 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.366058 21327 ts_tablet_manager.cc:1403] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.005s
I20251214 02:58:34.365990 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.367414 21248 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.368139 21432 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.368997 21327 raft_consensus.cc:359] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.369638 21327 raft_consensus.cc:385] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.369886 21327 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.364966 21639 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.370479 21327 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.372417 21323 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.372568 21327 ts_tablet_manager.cc:1434] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20251214 02:58:34.372859 21323 ts_tablet_manager.cc:1403] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.013s
I20251214 02:58:34.373464 21327 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:34.374854 21322 ts_tablet_manager.cc:1403] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.005s
I20251214 02:58:34.375543 21323 raft_consensus.cc:359] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.379350 21323 raft_consensus.cc:385] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.379609 21323 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.380193 21323 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.381661 21323 ts_tablet_manager.cc:1434] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20251214 02:58:34.383365 21322 raft_consensus.cc:359] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.384009 21322 raft_consensus.cc:385] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.384259 21322 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.384829 21322 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.386953 21322 ts_tablet_manager.cc:1434] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251214 02:58:34.387902 21323 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.378808 21327 tablet_bootstrap.cc:654] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.378474 21644 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.390906 21322 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.390959 21207 raft_consensus.cc:2393] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:34.365253 21437 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.399137 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.399720 21279 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.400269 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.401011 21207 raft_consensus.cc:2393] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:34.406137 21279 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:34.410355 21098 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.408314 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.406666 21437 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.407637 21644 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.414211 21649 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.416225 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.416239 21649 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:34.417068 21277 raft_consensus.cc:2393] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:34.417266 21649 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.431470 21322 tablet_bootstrap.cc:654] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.434895 21437 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:34.436437 21323 tablet_bootstrap.cc:654] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.439894 21327 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.440405 21327 ts_tablet_manager.cc:1403] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.015s
I20251214 02:58:34.452934 21323 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.451735 21327 raft_consensus.cc:359] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.453677 21327 raft_consensus.cc:385] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.454034 21327 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.453392 21323 ts_tablet_manager.cc:1403] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.007s
I20251214 02:58:34.457146 21173 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.457113 21323 raft_consensus.cc:359] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.458010 21323 raft_consensus.cc:385] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.458292 21323 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.458834 21644 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.459064 21323 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.461413 21323 ts_tablet_manager.cc:1434] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20251214 02:58:34.461488 21028 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:34.462832 21323 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.465237 21327 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.469039 21323 tablet_bootstrap.cc:654] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.469631 21437 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.470108 21437 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.471210 21327 ts_tablet_manager.cc:1434] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.031s user 0.010s sys 0.000s
I20251214 02:58:34.472132 21327 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:34.473392 21437 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.473971 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.479367 21322 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.479562 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.479812 21322 ts_tablet_manager.cc:1403] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.000s
I20251214 02:58:34.480144 21207 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.481281 21099 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.482120 21437 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.482638 21437 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.477079 21277 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.483027 21437 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.482147 21322 raft_consensus.cc:359] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.485066 21322 raft_consensus.cc:385] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.485332 21322 raft_consensus.cc:740] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.486274 21322 consensus_queue.cc:260] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.500110 21322 ts_tablet_manager.cc:1434] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251214 02:58:34.502517 21322 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.511478 21437 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.512995 21437 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.513917 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.514739 21207 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.520215 21327 tablet_bootstrap.cc:654] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.521299 21207 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:34.522464 21099 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.524142 21649 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.524616 21649 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:34.525425 21649 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.526108 21322 tablet_bootstrap.cc:654] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.528263 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.528716 21644 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.528909 21279 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.529207 21644 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.531211 21644 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.532791 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.533546 21132 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:34.533265 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.535591 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.536552 21644 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.541728 21645 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.542196 21645 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.544027 21645 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.545562 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.546458 21276 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.545614 21277 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.548101 21173 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:34.548862 21645 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.549196 21645 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.549511 21645 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.551873 21644 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.556383 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.565285 21322 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.565752 21322 ts_tablet_manager.cc:1403] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.005s
I20251214 02:58:34.570876 21322 raft_consensus.cc:359] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.571442 21322 raft_consensus.cc:385] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.571616 21322 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.566833 21127 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.572022 21322 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.578981 21327 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.579432 21327 ts_tablet_manager.cc:1403] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.108s user 0.015s sys 0.004s
I20251214 02:58:34.584066 21029 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:34.588155 21322 ts_tablet_manager.cc:1434] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20251214 02:58:34.584805 21432 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.590727 21432 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.592545 21432 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.594705 21323 tablet_bootstrap.cc:492] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:34.595091 21323 ts_tablet_manager.cc:1403] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.132s user 0.024s sys 0.017s
I20251214 02:58:34.595111 21322 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.604804 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:34.606316 21248 leader_election.cc:343] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.601914 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:34.609668 21247 leader_election.cc:343] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.610109 21247 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.611438 21432 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:34.613183 21645 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.615554 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.616113 21133 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.616539 21645 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.615820 21327 raft_consensus.cc:359] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.618925 21327 raft_consensus.cc:385] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.619202 21327 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.602169 21279 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:34.624464 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.624994 21276 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.625380 21652 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:34.631788 21323 raft_consensus.cc:359] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.632468 21323 raft_consensus.cc:385] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.632731 21323 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.633869 21323 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.635421 21133 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:34.619650 21327 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.644134 21432 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.645810 21173 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.646870 21432 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.647289 21323 ts_tablet_manager.cc:1434] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20251214 02:58:34.648751 21323 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:34.648953 21432 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.652083 21645 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.654104 21327 ts_tablet_manager.cc:1434] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.073s user 0.005s sys 0.002s
I20251214 02:58:34.656169 21327 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:34.653878 21645 raft_consensus.cc:697] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:34.660703 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.660897 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.661435 21207 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
W20251214 02:58:34.663007 21247 leader_election.cc:343] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.663566 21323 tablet_bootstrap.cc:654] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.663865 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.672168 21656 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.672668 21656 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.673087 21583 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.673467 21583 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.673799 21583 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.712064 21437 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.712599 21437 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.714855 21437 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.715426 21322 tablet_bootstrap.cc:654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.734167 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.735539 21659 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.736043 21659 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.738059 21659 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.738245 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.738888 21279 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.741926 21432 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:34.745007 21099 leader_election.cc:343] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.744939 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.731356 21649 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.756073 21098 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.756661 21437 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.757026 21437 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.757349 21437 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:34.764351 21099 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.742448 21432 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.740345 21645 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.772056 21327 tablet_bootstrap.cc:654] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.767134 21276 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:34.773080 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.774467 21639 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:34.755991 21649 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.777850 21432 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.764746 21656 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.785641 21437 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.787648 21437 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:34.791245 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43906.7us
I20251214 02:58:34.780891 21127 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.801568 21173 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.806660 21693 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.807066 21693 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.807380 21693 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.812131 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.812757 21278 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:34.798247 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.819370 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:34.823530 21276 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.801009 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.824640 21649 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.814783 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:34.826031 21099 leader_election.cc:343] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.795637 21652 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20251214 02:58:34.830422 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.838730 21437 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.839073 21437 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.839351 21437 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:34.843344 21248 leader_election.cc:343] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.859541 21437 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.804646 21583 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.871462 21322 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:34.871948 21322 ts_tablet_manager.cc:1403] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.283s user 0.023s sys 0.004s
I20251214 02:58:34.872192 21437 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.870705 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:34.873178 21133 raft_consensus.cc:2393] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:34.870913 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.874094 21276 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.874827 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.876268 21693 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:34.877223 21099 leader_election.cc:343] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.878180 21693 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:34.878887 21204 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:34.879181 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:34.879698 21279 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:34.884907 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.885421 21279 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.886329 21322 raft_consensus.cc:359] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.887013 21322 raft_consensus.cc:385] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:34.887256 21322 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:34.887852 21322 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.891994 21322 ts_tablet_manager.cc:1434] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20251214 02:58:34.891767 21247 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.895371 21322 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:34.892570 21098 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:34.895015 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:34.893630 21583 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.896261 21201 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.896750 21649 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:34.897112 21649 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:34.897419 21649 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.905730 21322 tablet_bootstrap.cc:654] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:34.919775 21649 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.909516 21690 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:34.909799 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.912147 21204 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.912748 21276 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:34.922683 21689 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 02:58:34.919601 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 59713.6us
I20251214 02:58:34.926252 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:34.926421 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:34.927059 21277 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:34.920459 21327 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:34.920769 21279 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:34.936364 21327 ts_tablet_manager.cc:1403] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.280s user 0.017s sys 0.017s
I20251214 02:58:34.908680 21201 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:34.938441 21098 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:34.960325 21647 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.960891 21647 raft_consensus.cc:697] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:34.961457 21277 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:34.943710 21689 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:34.938431 21247 leader_election.cc:343] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:34.963338 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:34.961709 21647 consensus_queue.cc:237] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.964805 21173 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:34.966001 21644 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.968605 21432 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:34.977531 21131 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:34.977639 21644 raft_consensus.cc:697] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:34.962224 21689 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.981010 21689 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:34.969105 21432 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:34.988981 21432 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:34.992069 21644 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:34.999143 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 60801.5us
I20251214 02:58:34.981878 21690 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:34.986487 21649 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.000984 21327 raft_consensus.cc:359] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.000563 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:35.015113 21327 raft_consensus.cc:385] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.015821 21327 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.015429 21657 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:35.016706 21657 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:35.017532 21657 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.019909 21327 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.021023 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:35.015846 21323 tablet_bootstrap.cc:492] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.029824 21323 ts_tablet_manager.cc:1403] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.381s user 0.025s sys 0.026s
I20251214 02:58:35.028378 21327 ts_tablet_manager.cc:1434] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.092s user 0.004s sys 0.003s
I20251214 02:58:35.022550 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.023132 21690 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:35.042102 21248 leader_election.cc:343] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.038954 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:35.052075 21279 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:35.051486 21327 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:35.054878 21437 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.045732 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.074124 21690 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.079150 21322 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.079643 21322 ts_tablet_manager.cc:1403] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.184s user 0.041s sys 0.014s
I20251214 02:58:35.087394 21656 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:35.088742 21710 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.092692 21323 raft_consensus.cc:359] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.093376 21323 raft_consensus.cc:385] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.093648 21323 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.094272 21323 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.146322 21327 tablet_bootstrap.cc:654] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.171203 21322 raft_consensus.cc:359] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.172171 21322 raft_consensus.cc:385] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 02:58:35.172400 21706 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.081s user 0.002s sys 0.000s
I20251214 02:58:35.172518 21322 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
W20251214 02:58:35.173566 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 76503.8us
I20251214 02:58:35.173435 21322 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.174453 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:35.176023 21248 leader_election.cc:343] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.177644 21322 ts_tablet_manager.cc:1434] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.098s user 0.015s sys 0.018s
I20251214 02:58:35.178599 21323 ts_tablet_manager.cc:1434] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.128s user 0.007s sys 0.000s
I20251214 02:58:35.178650 21322 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.179659 21323 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.181602 21027 catalog_manager.cc:5654] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:35.186051 21323 tablet_bootstrap.cc:654] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.186254 21322 tablet_bootstrap.cc:654] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.192790 21029 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:58:35.197539 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 109159us
I20251214 02:58:35.226219 21098 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:35.227963 21657 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.210821 21688 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.230084 21688 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.232043 21688 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.233819 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.213505 21327 tablet_bootstrap.cc:492] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:35.228570 21657 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.234512 21127 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
W20251214 02:58:35.229490 21247 leader_election.cc:343] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.234994 21327 ts_tablet_manager.cc:1403] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.184s user 0.013s sys 0.002s
I20251214 02:58:35.235051 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.236924 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.237823 21327 raft_consensus.cc:359] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.238512 21327 raft_consensus.cc:385] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.238791 21327 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.239331 21327 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.240368 21689 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.241065 21699 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.248101 21327 ts_tablet_manager.cc:1434] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20251214 02:58:35.249717 21327 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:35.252790 21247 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:35.252936 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.254408 21690 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:35.254796 21690 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.255103 21690 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.249992 21028 catalog_manager.cc:5654] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:35.259053 21699 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:35.259667 21248 leader_election.cc:343] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.259521 21699 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.259771 21028 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:35.261468 21699 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.263698 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.264392 21279 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:35.266312 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.273353 21322 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.273818 21322 ts_tablet_manager.cc:1403] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.003s
I20251214 02:58:35.275043 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.276460 21699 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.276332 21322 raft_consensus.cc:359] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.278700 21322 raft_consensus.cc:385] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.280583 21322 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.282414 21690 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.290094 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:58:35.291334 21248 leader_election.cc:343] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.281280 21322 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:35.296124 21247 leader_election.cc:343] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.296595 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.297852 21029 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:35.301075 21688 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.301944 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:35.302328 21322 ts_tablet_manager.cc:1434] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20251214 02:58:35.302539 21133 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.303251 21322 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.305125 21690 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.309806 21327 tablet_bootstrap.cc:654] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.315987 21699 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.319571 21323 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.320277 21323 ts_tablet_manager.cc:1403] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.141s user 0.021s sys 0.012s
I20251214 02:58:35.325392 21323 raft_consensus.cc:359] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.326079 21323 raft_consensus.cc:385] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.326364 21323 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.328253 21323 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.330498 21133 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.334985 21322 tablet_bootstrap.cc:654] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.344244 21247 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:35.345476 21688 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:35.345960 21688 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:58:35.337328 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:35.346741 21688 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.352388 21323 ts_tablet_manager.cc:1434] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20251214 02:58:35.353440 21323 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.359314 21322 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.359913 21322 ts_tablet_manager.cc:1403] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.005s
I20251214 02:58:35.359213 21709 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.363045 21323 tablet_bootstrap.cc:654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.363250 21709 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.363894 21644 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.366000 21709 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.362375 21322 raft_consensus.cc:359] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.366907 21322 raft_consensus.cc:385] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.365940 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.367235 21322 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.367727 21207 raft_consensus.cc:2393] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:35.367947 21322 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.371040 21327 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:35.371471 21327 ts_tablet_manager.cc:1403] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.007s
I20251214 02:58:35.378695 21323 tablet_bootstrap.cc:492] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.379259 21323 ts_tablet_manager.cc:1403] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20251214 02:58:35.382088 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.384037 21322 ts_tablet_manager.cc:1434] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20251214 02:58:35.385000 21322 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.386543 21098 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:35.387351 21708 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.388680 21327 raft_consensus.cc:359] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.385001 21323 raft_consensus.cc:359] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.391281 21323 raft_consensus.cc:385] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.390594 21327 raft_consensus.cc:385] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.393360 21327 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.395522 21699 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.391620 21323 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.399139 21323 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.401355 21323 ts_tablet_manager.cc:1434] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20251214 02:58:35.403111 21322 tablet_bootstrap.cc:654] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.407330 21323 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.413010 21323 tablet_bootstrap.cc:654] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.418605 21028 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:35.424285 21708 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.425191 21322 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.425771 21322 ts_tablet_manager.cc:1403] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.000s
I20251214 02:58:35.428367 21322 raft_consensus.cc:359] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.431164 21322 raft_consensus.cc:385] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.394412 21327 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.431638 21322 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.432783 21322 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.433869 21327 ts_tablet_manager.cc:1434] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.062s user 0.009s sys 0.000s
W20251214 02:58:35.436717 21099 consensus_peers.cc:597] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 -> Peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Couldn't send request to peer 668ad4f30bd649c88a034bbab9f13b20. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:35.439719 21322 ts_tablet_manager.cc:1434] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251214 02:58:35.440618 21322 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.441339 21323 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.441790 21323 ts_tablet_manager.cc:1403] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.006s
I20251214 02:58:35.443485 21327 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:35.445405 21323 raft_consensus.cc:359] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.446060 21323 raft_consensus.cc:385] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.446308 21323 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.446889 21323 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.449791 21323 ts_tablet_manager.cc:1434] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251214 02:58:35.450448 21323 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.462361 21327 tablet_bootstrap.cc:654] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.481160 21645 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251214 02:58:35.492338 21323 tablet_bootstrap.cc:654] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.496186 21322 tablet_bootstrap.cc:654] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.496490 21690 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.497709 21690 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.501398 21690 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.506922 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.507732 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:35.511071 21247 leader_election.cc:343] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:35.514202 21248 leader_election.cc:343] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.514642 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.513800 21688 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 02:58:35.518463 21690 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.522971 21323 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.523788 21323 ts_tablet_manager.cc:1403] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.006s
I20251214 02:58:35.525760 21323 raft_consensus.cc:359] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.529210 21690 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.536872 21323 raft_consensus.cc:385] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.537230 21323 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.538007 21323 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.540257 21323 ts_tablet_manager.cc:1434] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20251214 02:58:35.532162 21322 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.541261 21322 ts_tablet_manager.cc:1403] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.101s user 0.015s sys 0.000s
I20251214 02:58:35.541343 21323 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.547364 21322 raft_consensus.cc:359] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.547992 21323 tablet_bootstrap.cc:654] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.548786 21322 raft_consensus.cc:385] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.549095 21322 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.549650 21322 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.551131 21639 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.551599 21639 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.552235 21322 ts_tablet_manager.cc:1434] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20251214 02:58:35.553133 21322 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.553580 21639 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.556505 21657 consensus_queue.cc:1048] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.558912 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.559551 21204 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:35.561116 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:35.562346 21323 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.563010 21688 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:35.563352 21688 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.563509 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.563681 21688 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:35.567374 21247 leader_election.cc:343] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.569640 21323 ts_tablet_manager.cc:1403] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.012s
I20251214 02:58:35.569763 21327 tablet_bootstrap.cc:492] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:35.574331 21327 ts_tablet_manager.cc:1403] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.135s user 0.010s sys 0.002s
I20251214 02:58:35.572880 21688 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.577298 21323 raft_consensus.cc:359] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.577975 21323 raft_consensus.cc:385] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.578222 21323 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.578822 21323 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.579975 21322 tablet_bootstrap.cc:654] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.584805 21327 raft_consensus.cc:359] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.585592 21327 raft_consensus.cc:385] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.585919 21327 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.586593 21327 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.588584 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:35.589219 21688 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:58:35.590420 21247 leader_election.cc:343] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.594689 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:35.595337 21207 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.608086 21327 ts_tablet_manager.cc:1434] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20251214 02:58:35.609086 21327 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:35.606197 21207 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.612478 21323 ts_tablet_manager.cc:1434] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20251214 02:58:35.617201 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.618150 21323 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.614820 21709 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.618197 21432 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:35.618780 21432 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:35.619565 21432 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.625798 21688 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.628242 21688 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.636101 21688 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.637526 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.638190 21127 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:35.641603 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:35.642432 21690 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:35.642812 21690 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.643126 21690 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.650425 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.652041 21204 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:35.652081 21708 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.652244 21690 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.654022 21709 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.655777 21322 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.656597 21322 ts_tablet_manager.cc:1403] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.017s
I20251214 02:58:35.658830 21323 tablet_bootstrap.cc:654] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.656128 21708 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.688997 21690 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.695220 21327 tablet_bootstrap.cc:654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.690848 21708 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.699818 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:35.700363 21127 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.693435 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.695073 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.690164 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:35.697930 21644 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.698133 21322 raft_consensus.cc:359] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.704192 21279 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:35.705094 21322 raft_consensus.cc:385] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.705435 21322 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.706406 21644 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.706282 21322 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.708348 21644 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.708690 21322 ts_tablet_manager.cc:1434] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20251214 02:58:35.709244 21204 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:35.709932 21322 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.710922 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.711504 21133 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:35.706291 21207 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.712993 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.713557 21279 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:35.714715 21127 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.718881 21099 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:35.733937 21207 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.717123 21173 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:35.732475 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:35.720276 21029 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:35.738410 21710 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:35.738474 21644 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:35.738415 21645 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.741251 21644 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.741643 21644 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.741369 21645 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.743311 21645 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.747236 21322 tablet_bootstrap.cc:654] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.739511 21688 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:35.749482 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.750028 21279 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:35.750914 21688 raft_consensus.cc:697] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:35.751837 21688 consensus_queue.cc:237] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.739677 21710 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.754155 21710 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.755571 21644 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.762020 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:35.764891 21710 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.766219 21133 raft_consensus.cc:2393] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:35.755659 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.768296 21127 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:35.777746 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.780938 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:35.781661 21207 raft_consensus.cc:2393] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:35.783653 21644 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.779944 21659 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.785883 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:35.786432 21279 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.794247 21279 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:35.784976 21710 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.795359 21173 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.796294 21699 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:35.796738 21699 raft_consensus.cc:697] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:35.797528 21699 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.799757 21659 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.800798 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:35.801476 21276 raft_consensus.cc:2393] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:35.804993 21323 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.805441 21323 ts_tablet_manager.cc:1403] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.004s
I20251214 02:58:35.818233 21327 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:35.817732 21323 raft_consensus.cc:359] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.818849 21327 ts_tablet_manager.cc:1403] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.210s user 0.010s sys 0.003s
I20251214 02:58:35.819393 21029 catalog_manager.cc:5654] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:35.821286 21327 raft_consensus.cc:359] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.822434 21327 raft_consensus.cc:385] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.824183 21322 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.824692 21322 ts_tablet_manager.cc:1403] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.115s user 0.013s sys 0.000s
I20251214 02:58:35.825938 21323 raft_consensus.cc:385] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.826231 21323 raft_consensus.cc:740] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.827399 21322 raft_consensus.cc:359] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.827138 21323 consensus_queue.cc:260] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.822785 21327 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.828981 21327 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.831220 21327 ts_tablet_manager.cc:1434] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251214 02:58:35.828332 21322 raft_consensus.cc:385] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.831665 21322 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.832351 21322 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.834358 21323 ts_tablet_manager.cc:1434] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251214 02:58:35.835307 21323 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:35.842738 21327 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:35.864725 21694 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.865162 21694 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.858803 21322 ts_tablet_manager.cc:1434] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20251214 02:58:35.867466 21322 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:35.867396 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.867695 21694 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:35.849222 21098 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:35.868288 21129 raft_consensus.cc:2393] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:35.869055 21710 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.871472 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:35.873996 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.875345 21644 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:35.883838 21688 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.884356 21688 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.884753 21323 tablet_bootstrap.cc:654] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.887621 21690 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.891243 21644 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.887403 21028 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:35.897728 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.898398 21208 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.899784 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:35.900573 21688 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.900635 21690 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:35.901136 21690 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:35.901448 21690 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:35.904374 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.905041 21129 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:35.908519 21688 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.909075 21688 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.924157 21639 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:35.924729 21639 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.928090 21639 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.909874 21690 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.935851 21690 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.935945 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:35.936601 21132 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:35.937491 21322 tablet_bootstrap.cc:654] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.942158 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:35.942597 21207 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:35.934701 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.946549 21327 tablet_bootstrap.cc:654] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:35.948890 21207 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:35.938656 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.951056 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:35.955224 21690 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:35.955693 21690 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:35.956439 21690 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.957595 21132 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:35.958135 21639 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:35.959332 21247 leader_election.cc:343] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.958596 21639 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:35.962766 21248 leader_election.cc:343] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.963116 21248 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.965484 21432 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.966758 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:35.968043 21247 leader_election.cc:343] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.968544 21688 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:35.972088 21323 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:35.972662 21323 ts_tablet_manager.cc:1403] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.138s user 0.017s sys 0.007s
I20251214 02:58:35.974135 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:35.976723 21322 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:35.977197 21322 ts_tablet_manager.cc:1403] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.015s
I20251214 02:58:35.977082 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.978279 21323 raft_consensus.cc:359] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.979010 21323 raft_consensus.cc:385] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.979272 21323 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:35.979899 21323 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.975764 21639 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:58:35.981489 21247 leader_election.cc:343] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
W20251214 02:58:35.982743 21248 leader_election.cc:343] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.983183 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:35.983915 21639 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:35.986063 21639 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:35.989660 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:35.989946 21322 raft_consensus.cc:359] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.990648 21322 raft_consensus.cc:385] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:35.990907 21322 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
W20251214 02:58:35.999312 21248 leader_election.cc:343] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:35.991622 21322 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:35.999774 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:36.003876 21323 ts_tablet_manager.cc:1434] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20251214 02:58:36.005048 21323 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:36.004431 21031 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 2, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:36.007022 21322 ts_tablet_manager.cc:1434] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20251214 02:58:36.007926 21322 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: Bootstrap starting.
I20251214 02:58:36.009656 21327 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:36.010111 21327 ts_tablet_manager.cc:1403] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.168s user 0.022s sys 0.000s
I20251214 02:58:36.003931 21690 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:36.025642 21327 raft_consensus.cc:359] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.027400 21327 raft_consensus.cc:385] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.027736 21327 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.028628 21327 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.031472 21327 ts_tablet_manager.cc:1434] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20251214 02:58:36.032322 21322 tablet_bootstrap.cc:654] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.032363 21327 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:36.050033 21327 tablet_bootstrap.cc:654] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.054363 21323 tablet_bootstrap.cc:654] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.057327 21694 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.057809 21694 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.059725 21694 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.061167 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:36.061345 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:36.061939 21279 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
W20251214 02:58:36.062429 21173 leader_election.cc:343] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:36.063195 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:36.063917 21644 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:36.064277 21644 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:36.064602 21644 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.074190 21327 tablet_bootstrap.cc:492] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:36.074704 21327 ts_tablet_manager.cc:1403] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.004s
I20251214 02:58:36.073999 21644 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.080621 21644 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.081308 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:36.083530 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:36.084111 21279 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.086293 21322 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: No bootstrap required, opened a new log
I20251214 02:58:36.086068 21327 raft_consensus.cc:359] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.086753 21322 ts_tablet_manager.cc:1403] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.012s
W20251214 02:58:36.086845 21173 leader_election.cc:343] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:36.088109 21323 tablet_bootstrap.cc:492] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:36.088550 21323 ts_tablet_manager.cc:1403] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.009s
I20251214 02:58:36.089581 21322 raft_consensus.cc:359] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.090224 21322 raft_consensus.cc:385] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.090505 21322 raft_consensus.cc:740] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.091234 21322 consensus_queue.cc:260] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.093282 21279 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:36.094301 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:36.095052 21694 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:36.095513 21694 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:36.096288 21694 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.096516 21323 raft_consensus.cc:359] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.097102 21323 raft_consensus.cc:385] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.097375 21323 raft_consensus.cc:740] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.097981 21323 consensus_queue.cc:260] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.086774 21327 raft_consensus.cc:385] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.098913 21327 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.099241 21322 ts_tablet_manager.cc:1434] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20251214 02:58:36.099506 21327 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.100306 21688 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:36.101734 21327 ts_tablet_manager.cc:1434] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20251214 02:58:36.103650 21323 ts_tablet_manager.cc:1434] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20251214 02:58:36.104552 21323 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:36.110077 21323 tablet_bootstrap.cc:654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.113412 21327 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:36.143455 21323 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:36.143932 21323 ts_tablet_manager.cc:1403] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.001s
I20251214 02:58:36.146423 21323 raft_consensus.cc:359] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.149657 21323 raft_consensus.cc:385] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.149945 21323 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.150530 21323 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.152921 21323 ts_tablet_manager.cc:1434] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251214 02:58:36.154095 21323 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:36.165697 21323 tablet_bootstrap.cc:654] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.169997 21327 tablet_bootstrap.cc:654] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.177390 21028 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:36.185395 21133 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.191533 21323 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:36.192019 21323 ts_tablet_manager.cc:1403] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.006s
I20251214 02:58:36.195348 21644 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.195840 21644 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.196553 21323 raft_consensus.cc:359] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.197194 21323 raft_consensus.cc:385] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.197470 21323 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.197728 21644 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.199448 21711 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.199617 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:36.198045 21323 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.199970 21711 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.200870 21129 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:36.201834 21711 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.202864 21323 ts_tablet_manager.cc:1434] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251214 02:58:36.203289 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:36.203754 21323 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:36.204046 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:36.203601 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:36.205152 21327 tablet_bootstrap.cc:492] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:36.204598 21277 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:36.207729 21327 ts_tablet_manager.cc:1403] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.094s user 0.025s sys 0.011s
I20251214 02:58:36.208762 21098 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:36.210019 21323 tablet_bootstrap.cc:654] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.209964 21710 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:36.210911 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:36.210489 21710 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:36.211467 21279 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:36.205089 21644 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:36.211730 21710 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.211982 21644 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:36.212363 21644 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.220405 21710 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.221673 21327 raft_consensus.cc:359] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.227347 21327 raft_consensus.cc:385] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:36.227656 21327 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:36.228425 21710 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.228238 21327 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.228848 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:36.230523 21327 ts_tablet_manager.cc:1434] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20251214 02:58:36.231936 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:36.233775 21098 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:36.231995 21327 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:36.237064 21711 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:36.246591 21689 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:36.248365 21694 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.248926 21694 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.251652 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:36.251957 21694 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:36.253350 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:36.255156 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:36.256349 21688 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.257018 21688 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.258371 21645 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:36.266773 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:36.267359 21201 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:36.267809 21644 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:36.268471 21248 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:36.268910 21204 raft_consensus.cc:2393] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:36.420359 21690 raft_consensus.cc:2804] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:36.558604 21659 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:36.269255 21688 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:36.615733 21690 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:37.074113 21327 tablet_bootstrap.cc:654] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:36.852695 21690 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:36.731781 21659 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:37.114266 21247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.069s sys 0.098s
W20251214 02:58:37.115684 21247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.069s sys 0.098s
I20251214 02:58:37.133879 21323 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:37.134454 21323 ts_tablet_manager.cc:1403] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.931s user 0.035s sys 0.077s
W20251214 02:58:37.129658 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.000s sys 0.043s
W20251214 02:58:37.152521 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s user 0.000s sys 0.044s
I20251214 02:58:37.135588 21644 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:37.180325 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.021s sys 0.025s
W20251214 02:58:37.180755 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s user 0.021s sys 0.025s
I20251214 02:58:37.196133 21659 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.225203 21323 raft_consensus.cc:359] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.225939 21323 raft_consensus.cc:385] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:37.226179 21323 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:37.232672 21323 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.265316 21776 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.265913 21776 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.240268 21769 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:37.277240 21323 ts_tablet_manager.cc:1434] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
I20251214 02:58:37.278014 21323 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:37.281582 21689 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:37.279062 21769 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.282056 21689 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.289086 21782 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.289625 21782 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.291671 21782 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.301592 21327 tablet_bootstrap.cc:492] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:37.302058 21327 ts_tablet_manager.cc:1403] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 1.070s user 0.009s sys 0.013s
I20251214 02:58:37.287403 21780 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.305174 21783 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.305706 21783 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.306198 21769 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:37.307797 21783 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.307225 21771 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.295162 21776 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:37.310487 21780 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.323536 21327 raft_consensus.cc:359] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.324772 21327 raft_consensus.cc:385] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:37.325049 21327 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:37.325404 21780 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.310338 21771 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.345538 21771 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:37.377599 21772 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.378266 21772 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:37.703377 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 380342us
I20251214 02:58:37.377642 21773 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.801580 21774 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.811067 21774 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.813259 21774 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.813864 21773 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.813321 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:37.801412 21767 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.836449 21778 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.836611 21774 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.805698 21780 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.803964 21782 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.837226 21774 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.837030 21778 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.846473 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 183) took 1661 ms. Trace:
I20251214 02:58:37.839946 21767 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.832069 21785 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.832794 21201 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:37.846947 21133 rpcz_store.cc:276] 1214 02:58:36.184830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:58:36.185007 (+ 177us) service_pool.cc:225] Handling call
1214 02:58:36.185279 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:58:37.846165 (+1660886us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:58:37.846390 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:58:37.850234 21772 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.850728 21772 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.851145 21772 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.852929 21772 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.853408 21772 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.853842 21772 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.855062 21773 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.855618 21773 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.856082 21773 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.857964 21773 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.841037 21783 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:37.858913 21783 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.860795 21783 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.858408 21773 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.867517 21779 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.867781 21773 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.868016 21779 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.842685 21780 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.844682 21782 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.872715 21779 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.886406 21779 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.881928 21778 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.886883 21779 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.882037 21767 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.882707 21789 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.887439 21778 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.887789 21789 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.887967 21778 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:37.889122 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.080s user 0.037s sys 0.056s
I20251214 02:58:37.883822 21774 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.885810 21780 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:37.889415 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.080s user 0.037s sys 0.056s
I20251214 02:58:37.864655 21785 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:37.890806 21767 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.902164 21790 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.902252 21767 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.892223 21782 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.894698 21779 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.895231 21778 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.895525 21789 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:37.896603 21204 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 241) took 1693 ms (client timeout 1941 ms). Trace:
I20251214 02:58:37.896935 21773 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.897953 21772 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.903455 21785 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:37.898342 21784 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:58:37.905073 21129 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.647s user 0.016s sys 0.059s
W20251214 02:58:37.913818 21129 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.656s user 0.016s sys 0.060s
W20251214 02:58:37.910943 21204 rpcz_store.cc:269] 1214 02:58:36.202900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:58:36.203037 (+ 137us) service_pool.cc:225] Handling call
1214 02:58:37.896374 (+1693337us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b540037b4d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:58:37.896509 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124101376}
W20251214 02:58:37.914822 21172 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.351s user 0.005s sys 0.038s
W20251214 02:58:37.915120 21172 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.352s user 0.005s sys 0.038s
I20251214 02:58:37.913888 21784 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.917989 21772 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.918215 21767 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.913112 21773 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.902776 21790 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.919322 21772 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.919150 21773 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.919878 21784 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.920532 21790 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.923345 21773 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.923892 21772 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.957656 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 186) took 1701 ms. Trace:
I20251214 02:58:37.964192 21767 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.958786 21129 rpcz_store.cc:276] 1214 02:58:36.256010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:58:36.256687 (+ 677us) service_pool.cc:225] Handling call
1214 02:58:36.256997 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:58:36.257285 (+ 288us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:58:36.257312 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:58:36.257775 (+ 463us) log.cc:844] Serialized 26 byte log entry
1214 02:58:37.956434 (+1698659us) raft_consensus.cc:1645] Marking committed up to 0
1214 02:58:37.956604 (+ 170us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:58:37.956789 (+ 185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:58:37.956903 (+ 114us) raft_consensus.cc:1685] finished
1214 02:58:37.956936 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:58:37.957269 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1021,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":56448,"thread_start_us":1656116,"threads_started":1,"wal-append.queue_time_us":812200}
I20251214 02:58:37.965454 21767 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.968719 21767 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.969312 21767 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.970043 21767 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.972282 21767 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.972765 21767 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.973534 21767 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.974889 21772 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.975680 21772 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.977963 21772 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.978477 21772 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.979228 21772 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.981251 21772 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.981738 21772 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.982926 21773 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:37.983982 21796 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.985430 21778 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.985612 21779 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.986241 21778 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.986464 21779 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.983747 21773 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.989938 21794 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 02:58:37.992231 21784 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.992928 21784 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.997718 21782 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:37.999017 21779 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.999663 21778 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.000700 21784 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:37.998443 21782 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.001356 21797 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:37.982369 21772 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.003159 21782 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:38.003649 21171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.386s user 0.012s sys 0.043s
W20251214 02:58:38.004473 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.389s user 0.013s sys 0.044s
I20251214 02:58:37.999614 21780 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:37.987491 21796 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.004122 21794 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.007097 21782 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.007782 21772 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.008208 21773 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.004218 21784 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.008354 21780 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.011586 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.012013 21783 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.012298 21772 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.012770 21783 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.013146 21780 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.010178 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.010035 21794 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.013010 21772 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.016620 21794 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.017203 21794 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.017439 21772 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.011523 21795 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.018826 21784 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.020684 21204 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.021349 21773 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.022473 21784 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.022738 21767 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.023308 21767 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.023442 21796 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.024247 21202 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251214 02:58:38.023933 21767 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.022082 21773 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.002118 21797 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.017786 21794 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.015292 21783 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.017786 21782 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.029085 21773 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.028241 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.030181 21794 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.030843 21782 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.030893 21767 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.032344 21797 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.033735 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.025022 21795 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.035574 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.041026 21204 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.041589 21782 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.035953 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.044209 21133 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:38.044596 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.030123 21129 raft_consensus.cc:2393] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:38.043190 21782 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.047838 21782 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.049072 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.049503 21782 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.049752 21204 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:38.050258 21782 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.050799 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.052927 21782 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.053076 21133 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.054426 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.055159 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.056203 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.063714 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.064448 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.064620 21202 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.065943 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.069026 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.069947 21205 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.070516 21204 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.049762 21795 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.068077 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.073794 21797 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.076808 21797 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.067689 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.079670 21796 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.080370 21796 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.082932 21796 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.083751 21772 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.084239 21202 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.085388 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.085592 21783 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.086200 21131 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.087626 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.087688 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.088362 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.089437 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.090322 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.086210 21783 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.092912 21783 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.084362 21772 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.095674 21772 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.098517 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.103029 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.104804 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.105549 21797 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.102942 21785 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.106305 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.106396 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.089318 21784 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.106774 21785 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.107407 21784 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.107934 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.108752 21785 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.110615 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.108737 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.111056 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.111312 21208 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.112947 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:38.114094 21245 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.033s sys 0.034s
I20251214 02:58:38.114713 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.115175 21199 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.115253 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:38.114477 21245 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s user 0.033s sys 0.034s
I20251214 02:58:38.109200 21784 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.124604 21765 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.124799 21787 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.124933 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.125334 21765 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.126282 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.126997 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.123554 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.130228 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.126371 21786 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.131175 21786 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.116631 21205 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.125432 21787 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.144269 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.146807 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.147356 21202 raft_consensus.cc:2410] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251214 02:58:38.151252 21786 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.152405 21765 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.156404 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.147511 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.148211 21787 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.158818 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.161398 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.160980 21130 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251214 02:58:38.159657 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.162732 21125 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.168111 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.169436 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.169814 21782 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.170149 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.170230 21782 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.170724 21782 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.171169 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.160894 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.178431 21132 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.180388 21765 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.181023 21765 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.191962 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.192371 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.193352 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.193603 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.156426 21689 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.194878 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.196712 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.197455 21765 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.197453 21125 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.198204 21694 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:38.196667 21127 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.202378 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.191860 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.197758 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.203244 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.205086 21132 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
W20251214 02:58:38.207847 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s user 0.010s sys 0.005s
W20251214 02:58:38.208220 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.891s user 0.010s sys 0.005s
I20251214 02:58:38.215426 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.216843 21778 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.217123 21099 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.217234 21778 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.217716 21778 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.218982 21785 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:38.219360 21785 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:38.219746 21785 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:38.200717 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.203840 21689 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.224400 21689 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.220038 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.223486 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.227782 21797 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:38.228307 21797 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:38.228652 21797 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.228621 21126 raft_consensus.cc:2410] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251214 02:58:38.229840 21799 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.231937 21799 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.235342 21800 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.241449 21799 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.243914 21778 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.245834 21778 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.236763 21800 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.250972 21689 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.251235 21800 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.252122 21770 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.259788 21689 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.263152 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.263043 21323 tablet_bootstrap.cc:654] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:38.261967 21689 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.274454 21659 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.274971 21659 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.275332 21659 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.275883 21772 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:38.263608 21800 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.276495 21765 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.282915 21800 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.283691 21765 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.285408 21800 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.285689 21765 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.252799 21770 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.285859 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.292058 21688 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.293411 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.281322 21782 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.298974 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 198) took 1047 ms. Trace:
I20251214 02:58:38.300776 21799 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.291347 21772 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.300650 21785 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.302203 21689 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.292660 21688 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:37.325615 21327 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.295286 21099 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.304867 21207 rpcz_store.cc:276] 1214 02:58:37.251797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:58:38.297927 (+1046130us) service_pool.cc:225] Handling call
1214 02:58:38.298323 (+ 396us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:58:38.298526 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:58:38.298582 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:58:38.298624 (+ 42us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:58:38.298652 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:58:38.298706 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:58:38.298815 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:58:38.309159 21765 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.312302 21799 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.313992 21130 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:38.312419 21765 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.315479 21327 ts_tablet_manager.cc:1434] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 1.013s user 0.010s sys 0.003s
W20251214 02:58:38.315835 21327 ts_tablet_manager.cc:1467] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Tablet startup took 1013ms
I20251214 02:58:38.316211 21765 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.317014 21799 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.317795 21784 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.318289 21784 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.319036 21782 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.308323 21772 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:58:38.316138 21327 ts_tablet_manager.cc:1469] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Trace:
1214 02:58:27.875619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:58:27.991305 (+115686us) service_pool.cc:225] Handling call
1214 02:58:28.055815 (+ 64510us) spinlock_profiling.cc:243] Waited 8.33 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb30f2 00007f6714d84419 00007f6713edd108 00007f6713f9f890 00007f6713f4e4c9 00007f670d3e6bf4 00007f670d3e6b82 00007f670d3e6ab1 00007f670d3e6a2c
1214 02:58:28.062632 (+ 6817us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1214 02:58:28.062746 (+ 114us) ts_tablet_manager.cc:682] Creating new metadata...
1214 02:58:28.210702 (+147956us) tablet_metadata.cc:641] Metadata flushed
1214 02:58:28.219878 (+ 9176us) tablet_replica.cc:206] Creating consensus instance
1214 02:58:28.220284 (+ 406us) inbound_call.cc:173] Queueing success response
1214 02:58:36.231702 (+8011418us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1214 02:58:37.302315 (+1070613us) ts_tablet_manager.cc:1435] Starting tablet replica
1214 02:58:37.302870 (+ 555us) tablet_replica.cc:261] Starting instrumentation
1214 02:58:37.303454 (+ 584us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":1278,"raft.queue_time_us":6269,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":156,"spinlock_wait_cycles":19158272,"tablet-open.queue_time_us":8011490}
I20251214 02:58:38.320812 21772 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:58:38.309770 21173 leader_election.cc:343] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.329455 21099 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:38.332046 21229 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0000 after lost signal to thread 19984
I20251214 02:58:38.332751 21173 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:38.333083 21229 debug-util.cc:398] Leaking SignalData structure 0x7b08000f63e0 after lost signal to thread 21065
W20251214 02:58:38.328825 21690 consensus_meta.cc:287] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 1.244s user 0.011s sys 0.016s
I20251214 02:58:38.333609 21690 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:38.334189 21229 debug-util.cc:398] Leaking SignalData structure 0x7b0800111700 after lost signal to thread 21157
I20251214 02:58:38.334736 21689 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.334919 21099 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.335330 21689 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:38.336424 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:38.336755 21690 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.337203 21689 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.337669 21099 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.345062 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.340583 21699 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.300650 21659 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.355113 21802 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.360198 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.338315 21790 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.329113 21778 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.363941 21644 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.364419 21644 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.364161 21694 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.366133 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.323824 21784 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.364811 21644 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.340440 21765 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.310083 21800 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.363152 21790 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.340350 21688 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.358875 21782 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.338634 21787 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.340638 21799 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.338269 21797 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.373162 21792 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.367161 21694 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.374858 21323 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:38.360419 21699 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.364109 21804 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.380733 21765 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.382314 21787 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.361310 21803 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.381593 21099 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.361105 21798 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.363955 21689 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.367053 21778 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.380393 21790 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:58:38.381768 21229 debug-util.cc:398] Leaking SignalData structure 0x7b08000e48c0 after lost signal to thread 21232
I20251214 02:58:38.381417 21782 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.397221 21798 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.397775 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.398212 21785 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.382069 21802 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.398905 21803 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.404448 21793 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.407572 21127 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:38.383558 21688 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.393957 21792 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.399406 21765 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:58:38.403029 21229 debug-util.cc:398] Leaking SignalData structure 0x7b080013bb00 after lost signal to thread 21233
I20251214 02:58:38.391594 21694 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.390338 21699 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.412196 21323 ts_tablet_manager.cc:1403] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 1.134s user 0.023s sys 0.037s
I20251214 02:58:38.407933 21798 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.409855 21770 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.406255 21787 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.408638 21799 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.403560 21778 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.384512 21690 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.390867 21800 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.406601 21782 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.434726 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.417982 21784 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.425666 21792 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.430838 21793 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.389854 21804 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.450201 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.426873 21805 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.448024 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.431927 21765 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.443156 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.431244 21659 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.450431 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.452782 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.452900 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.453125 21803 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.453536 21639 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.425882 21689 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.441857 21128 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.442217 21797 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.440853 21784 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.448838 21772 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.442551 21787 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.453621 21805 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.460402 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.457440 21327 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:38.461417 21802 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.461856 21789 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.462081 21689 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.463632 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.437948 21688 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.478314 21789 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:38.478772 21789 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.449622 21793 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.459774 21323 raft_consensus.cc:359] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.479998 21323 raft_consensus.cc:385] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:38.438408 21644 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.474200 21779 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.475414 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.476413 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.477203 21432 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:38.480413 21323 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:38.459239 21765 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.479965 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.483872 21323 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.488662 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:58:38.479404 21229 debug-util.cc:398] Leaking SignalData structure 0x7b080025c8e0 after lost signal to thread 21306
I20251214 02:58:38.467067 21784 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.492691 21799 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.456178 21690 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.485343 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.498737 21432 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.501979 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.499337 21800 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.499234 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.491161 21779 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.501668 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.515247 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.505977 21688 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.521772 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.499603 21639 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.506304 21799 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.504257 21787 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.545545 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.515040 21786 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20251214 02:58:38.568220 21644 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.567730 21699 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.541797 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.572402 21323 ts_tablet_manager.cc:1434] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.139s user 0.006s sys 0.001s
I20251214 02:58:38.547122 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.575766 21204 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:38.541348 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.578722 21786 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:38.580590 21786 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:58:38.581077 21779 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.582266 21099 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.514007 21808 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.520629 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.538869 21811 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.584209 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.544898 21432 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.563303 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.562028 21128 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.589888 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.580449 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.591228 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.592794 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.593967 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.596599 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.599987 21765 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.599320 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.567215 21799 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.600150 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.587843 21783 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:38.598654 21199 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:38.584805 21811 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.606709 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.607239 21202 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:38.607700 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.594482 21772 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.609081 21794 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:38.608777 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.596323 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.617302 21811 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.625077 21804 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.624845 21805 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.603982 21783 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:38.637778 21783 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.603549 21778 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:38.616915 21247 leader_election.cc:343] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.603245 21699 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.605341 21690 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.661805 21786 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.646080 21790 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.659855 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:38.663930 21276 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.664124 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.657647 21789 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.654021 21794 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:58:38.661762 21812 log.cc:927] Time spent T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20251214 02:58:38.665946 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.670782 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.671617 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.672205 21200 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.671468 21205 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:38.674785 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.675158 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.675859 21282 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:38.676373 21778 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.669230 21699 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.677802 21699 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.676542 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.675266 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.664140 21790 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.683151 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.683740 21205 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.683984 21323 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:38.683987 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.683904 21203 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:58:38.660730 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 99306.5us
I20251214 02:58:38.659924 21694 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.658474 21798 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.687263 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.678596 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.680853 21787 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.682255 21639 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.684957 21202 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:38.700237 21207 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.704281 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.704298 21203 raft_consensus.cc:2393] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:38.704928 21282 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.689087 21789 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.699103 21784 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.708734 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.709313 21206 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.709514 21784 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.604660 21808 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.712760 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.716133 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.720216 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.720800 21279 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.721127 21327 tablet_bootstrap.cc:654] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:38.711650 21799 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.724205 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:38.725625 21813 log.cc:927] Time spent T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251214 02:58:38.733390 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.735742 21127 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.723763 21206 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:38.728154 21694 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.748605 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.749342 21282 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.749869 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.749253 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.728049 21793 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.755234 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.756464 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:38.759716 21173 leader_election.cc:343] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.757573 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.761438 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.761644 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.764070 21787 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:38.764562 21808 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.765162 21280 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.747594 21199 raft_consensus.cc:1240] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.772723 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:38.761502 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.732307 21794 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.777208 21124 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.779098 21278 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.768064 21699 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.783231 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.763471 21782 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.789029 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.789161 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.788147 21639 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:38.790714 21639 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:38.791049 21639 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.791925 21276 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
W20251214 02:58:38.795787 21247 leader_election.cc:343] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.786130 21798 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.815977 21205 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:38.795647 21782 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:38.819983 21247 leader_election.cc:343] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.819722 21783 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.763572 21792 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.833345 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.838797 21276 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.839282 21793 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.839733 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.841903 21792 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:38.843371 21247 leader_election.cc:343] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.835470 21783 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:38.846490 21127 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:38.845863 21130 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.852712 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.846932 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.847231 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.847795 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.855003 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.848786 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.849560 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.855952 21200 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:38.850138 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.856628 21207 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.850296 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.857088 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.851298 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.853507 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.860226 21125 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.861620 21323 tablet_bootstrap.cc:654] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:38.862375 21125 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251214 02:58:38.863960 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.865707 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.867532 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.871428 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.871519 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.873030 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.873174 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.873057 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.876231 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.877249 21277 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.879137 21798 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:38.873276 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.879703 21798 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:38.880235 21125 raft_consensus.cc:2393] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:38.879760 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.880638 21798 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.882102 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.882548 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.882731 21275 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.883327 21282 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.883538 21639 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.884361 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.885012 21276 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.885432 21639 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.886900 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.886920 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.887223 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:38.888978 21247 leader_election.cc:343] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:38.889111 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.890291 21639 consensus_queue.cc:1059] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:38.888639 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.892349 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.892702 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.892989 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.893604 21275 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.896117 21099 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.899416 21099 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.901335 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.909765 21279 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.901388 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.913038 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.911769 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.900012 21792 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:38.901818 21782 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:38.901654 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.915843 21779 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:38.917508 21099 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:38.918306 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.926791 21779 raft_consensus.cc:697] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:38.927874 21779 consensus_queue.cc:237] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.924981 21793 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:38.919945 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:38.934870 21274 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:38.918423 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.937290 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.937884 21205 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:38.927870 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:38.919835 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.919762 21792 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:38.955783 21125 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.940505 21782 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:38.952857 21639 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:38.952145 21793 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:38.963833 21327 tablet_bootstrap.cc:492] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:38.960556 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.964634 21327 ts_tablet_manager.cc:1403] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.529s user 0.015s sys 0.009s
I20251214 02:58:38.964731 21323 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:38.965133 21323 ts_tablet_manager.cc:1403] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.281s user 0.023s sys 0.004s
I20251214 02:58:38.957346 21792 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:38.971226 21205 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:38.956964 21782 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.973500 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:38.967767 21639 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.975139 21125 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:38.977224 21327 raft_consensus.cc:359] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.979291 21639 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:38.976629 21793 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.979979 21327 raft_consensus.cc:385] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:38.984558 21327 raft_consensus.cc:740] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:38.985673 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.985508 21327 consensus_queue.cc:260] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:38.993724 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:38.996074 21327 ts_tablet_manager.cc:1434] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20251214 02:58:38.997229 21327 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:38.999235 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.004906 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.006114 21323 raft_consensus.cc:359] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.006830 21323 raft_consensus.cc:385] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:39.007082 21323 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:39.007196 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.007526 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:39.000916 21099 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:39.014073 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.014966 21279 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:39.017092 21127 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20251214 02:58:39.011482 21323 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.016103 21808 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251214 02:58:39.021643 21132 raft_consensus.cc:1240] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.007234 21126 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.021672 21808 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.022855 21808 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.023586 21808 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.024757 21808 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.016392 21791 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:39.030812 21694 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:39.030718 21690 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:39.031323 21694 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.031774 21690 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.032658 21690 raft_consensus.cc:3055] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.032943 21690 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.033151 21694 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.033710 21690 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.012436 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:39.034727 21786 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.012125 21639 consensus_queue.cc:1059] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.009s
I20251214 02:58:39.038132 21786 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.058086 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:39.058759 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.061582 21786 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.062757 21786 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:58:39.062947 21173 leader_election.cc:343] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:39.059579 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.050503 21787 consensus_queue.cc:1059] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251214 02:58:39.050240 21791 raft_consensus.cc:697] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.067850 21805 raft_consensus.cc:3055] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.068202 21805 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.068517 21791 consensus_queue.cc:237] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.068935 21805 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.071058 21805 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.073213 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.072701 21323 ts_tablet_manager.cc:1434] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.107s user 0.009s sys 0.005s
I20251214 02:58:39.073771 21279 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:39.076089 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.076931 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.057150 21639 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251214 02:58:39.077714 21279 raft_consensus.cc:2393] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:39.078310 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.101897 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.081254 21327 tablet_bootstrap.cc:654] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:39.102345 21787 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.103168 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.103528 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.101073 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.103497 21787 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.104136 21799 raft_consensus.cc:3055] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.104404 21787 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.104895 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.105633 21277 raft_consensus.cc:2393] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:39.101020 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.072264 21690 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.078908 21639 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.110558 21804 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 02:58:39.113898 21804 consensus_queue.cc:1243] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer 721d604130924d939d291fe3eef8f6d1 log is divergent from this leader: its last log entry 3.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251214 02:58:39.115067 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.116297 21323 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Bootstrap starting.
I20251214 02:58:39.115013 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.118685 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.119321 21811 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:39.119705 21811 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.120077 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.120457 21811 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.121665 21811 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:39.104507 21799 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.123170 21799 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.122462 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.079196 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.124269 21799 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.130021 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.130800 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.105479 21787 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.135104 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.116210 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.138761 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.141376 21276 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:39.112936 21639 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.140705 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.164875 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.167899 21277 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.168638 21810 raft_consensus.cc:3055] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.167901 21787 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.169031 21810 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.167415 21323 tablet_bootstrap.cc:654] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:39.174503 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.174026 21031 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.175344 21275 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:39.178026 21278 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
W20251214 02:58:39.164714 21824 log.cc:927] Time spent T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20251214 02:58:39.185570 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.178934 21031 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.191268 21792 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.194972 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.195142 21792 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.203783 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.204275 21639 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.205662 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:58:39.208707 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:39.214537 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.215197 21273 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.197078 21277 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:39.222731 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.227663 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.230257 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.188241 21810 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.232782 21098 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.234376 21791 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.234930 21639 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.234486 21810 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.230930 21206 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.229208 21276 raft_consensus.cc:2393] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:58:39.248620 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:58:39.251938 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:39.250209 21279 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.262733 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.263898 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.264032 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.265770 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.266613 21810 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.266883 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.267509 21277 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:39.267663 21281 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.268877 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.269503 21282 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:39.271847 21274 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:39.275754 21098 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.279532 21276 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:39.280524 21792 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.281749 21098 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.282519 21792 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.285974 21206 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.287917 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.289218 21098 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.289698 21099 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:39.292375 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.292976 21814 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.293627 21814 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.293962 21814 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.294294 21814 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.294204 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.290643 21789 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.296737 21785 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.291191 21792 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.296975 21098 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.299610 21327 tablet_bootstrap.cc:492] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:39.300040 21327 ts_tablet_manager.cc:1403] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.303s user 0.010s sys 0.023s
I20251214 02:58:39.298641 21273 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.302098 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.302805 21805 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.297286 21789 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.303382 21323 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: No bootstrap required, opened a new log
I20251214 02:58:39.303640 21789 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.303843 21323 ts_tablet_manager.cc:1403] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Time spent bootstrapping tablet: real 0.188s user 0.030s sys 0.032s
I20251214 02:58:39.304558 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.307758 21792 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.307898 21785 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.308749 21792 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.309098 21814 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.309566 21327 raft_consensus.cc:359] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.310227 21327 raft_consensus.cc:385] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:39.310482 21327 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:39.311203 21248 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.311060 21327 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.312588 21248 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.313341 21248 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.307762 21805 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.314873 21639 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.316140 21323 raft_consensus.cc:359] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.316787 21323 raft_consensus.cc:385] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:39.317054 21323 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Initialized, Role: FOLLOWER
I20251214 02:58:39.317220 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.318719 21323 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.321187 21323 ts_tablet_manager.cc:1434] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251214 02:58:39.322587 21805 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.323805 21248 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.324607 21814 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.325050 21787 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.325271 21278 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.325542 21639 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.326510 21694 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.335534 21789 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.337453 21789 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.338167 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:58:39.326349 21248 leader_election.cc:343] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:39.329844 21098 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.339308 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.339839 21814 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.340224 21814 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.340703 21814 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.341121 21098 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.341322 21248 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.342309 21785 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.342746 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.343281 21098 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.343760 21787 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.331328 21327 ts_tablet_manager.cc:1434] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20251214 02:58:39.340175 21639 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.345028 21805 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.346287 21327 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:39.346593 21791 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.346992 21791 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.347347 21791 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.346292 21248 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.351593 21828 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:39.351823 21694 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:39.353120 21694 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.353008 21828 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.355953 21828 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.355906 21814 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.357630 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:39.358501 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.359596 21694 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.361285 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:39.366223 21787 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.367120 21814 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.366122 21248 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.366755 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.368292 21327 tablet_bootstrap.cc:654] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:39.369335 21098 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.367260 21787 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:39.370702 21248 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.370152 21805 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.371289 21639 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.371905 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.372089 21248 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.372944 21248 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.373144 21098 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:58:39.373796 21248 leader_election.cc:343] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): Illegal state: must be running to vote when last-logged opid is not known
I20251214 02:58:39.374205 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.374732 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.379338 21248 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.378644 21789 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.380244 21789 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.380038 21810 consensus_queue.cc:1059] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.380661 21789 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.373899 21805 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.380890 21799 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.380115 21793 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.381822 21793 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.375494 21639 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.382185 21793 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.370857 21814 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.375419 21785 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.370972 21787 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.384394 21811 raft_consensus.cc:2804] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:39.384684 21124 raft_consensus.cc:1240] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.384850 21811 raft_consensus.cc:697] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.385776 21811 consensus_queue.cc:237] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.387706 21810 raft_consensus.cc:3055] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.388099 21810 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.389866 21248 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.375223 21792 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.388906 21810 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.382776 21814 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.393433 21789 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.394524 21690 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.395377 21789 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.395659 21810 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.397778 21793 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.390275 21694 consensus_queue.cc:1059] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251214 02:58:39.399722 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.399921 21694 raft_consensus.cc:3055] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.407603 21694 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:39.408473 21694 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.409631 21694 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.399179 21814 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.400620 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.410362 21098 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.417475 21792 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.417896 21792 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.418274 21792 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:58:39.418980 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:39.419525 21124 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:39.421048 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.421108 21829 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.422223 21789 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.422582 21789 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.422920 21789 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.425132 21792 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.427729 21792 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.429590 21789 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.431190 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.431932 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.432305 21098 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.432972 21778 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.433537 21778 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.433748 21098 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.436610 21778 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.436801 21130 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.439420 21699 consensus_queue.cc:1059] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.443154 21828 raft_consensus.cc:3055] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.443495 21828 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:39.444240 21828 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.445148 21828 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.445623 21789 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.446321 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.447209 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.448244 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.449680 21778 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.450524 21098 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.458678 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:39.459774 21173 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.550s (SENT) [suppressed 1 similar messages]
I20251214 02:58:39.459236 21098 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.462005 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.462246 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.463219 21248 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.448246 21792 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.479604 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.480031 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.476001 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.404184 21791 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.483711 21279 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:39.485562 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.486044 21279 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.473240 21779 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.476419 21790 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.475103 21799 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.474889 21785 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.475013 21805 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.473093 21639 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.476227 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.507120 21127 raft_consensus.cc:1240] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.482530 21793 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.473062 21690 consensus_queue.cc:1059] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251214 02:58:39.498864 21783 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:39.498157 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.504767 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.508168 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.509969 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:58:39.515841 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56102.6us
I20251214 02:58:39.517736 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.533149 21787 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.542176 21279 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:39.522846 21829 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:39.550295 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.520364 21783 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.566869 21130 raft_consensus.cc:1240] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.571169 21125 raft_consensus.cc:1240] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.572360 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:39.569411 21779 raft_consensus.cc:697] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.575390 21779 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.576987 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.579200 21791 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.561033 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.580942 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.581990 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.580056 21279 raft_consensus.cc:3055] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.583876 21279 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.584533 21279 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.585268 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.586196 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:58:39.591915 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:39.586809 21789 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.592057 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.593194 21789 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.587800 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:58:39.555552 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:39.568290 21787 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:39.594424 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.591586 21098 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.596166 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.597136 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:39.598232 21098 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.599936 21792 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.601150 21792 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.601576 21792 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.605118 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.582384 21814 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.606160 21202 raft_consensus.cc:2393] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:39.607007 21098 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.607020 21794 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.608048 21794 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:39.556444 21829 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.608827 21794 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.611105 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.611439 21783 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.612057 21783 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.612411 21281 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 1. (term mismatch)
I20251214 02:58:39.612813 21281 pending_rounds.cc:85] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 0
I20251214 02:58:39.613080 21281 pending_rounds.cc:107] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20251214 02:58:39.613355 21281 raft_consensus.cc:2889] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:58:39.613569 21783 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.616041 21199 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.605521 21782 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.623517 21794 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.624589 21814 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.625622 21794 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.602421 21792 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:39.627691 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 58301.9us
I20251214 02:58:39.628543 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.629313 21794 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:58:39.630152 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:39.615983 21829 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.629774 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.636979 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.637283 21790 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.637709 21208 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:39.638213 21790 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.639609 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.640515 21829 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.640892 21829 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:39.641287 21829 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.645018 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.650190 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.650993 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.662700 21699 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.663853 21203 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 0.
I20251214 02:58:39.664072 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.667161 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.668802 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.669613 21203 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:39.671293 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.672878 21829 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.676468 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.676750 21127 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.676993 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.677793 21790 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.677919 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.679198 21203 raft_consensus.cc:1240] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.679641 21790 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.680256 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.681272 21207 raft_consensus.cc:1240] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.681337 21248 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.681931 21206 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20251214 02:58:39.682767 21275 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:39.684015 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.684762 21206 raft_consensus.cc:2393] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:39.684677 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.686053 21786 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.684756 21279 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.686779 21786 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.682682 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.688902 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.687845 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.692075 21785 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.692580 21804 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.692977 21785 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.693498 21279 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:58:39.694132 21639 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20251214 02:58:39.695071 21248 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.691638 21028 catalog_manager.cc:5654] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.696183 21639 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.699707 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.702608 21699 raft_consensus.cc:2764] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20251214 02:58:39.703624 21794 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.699956 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.704535 21829 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.705123 21790 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:39.707386 21790 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.709165 21790 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.709394 21204 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:39.710156 21099 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.710976 21851 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.696554 21028 catalog_manager.cc:5654] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.720011 21028 catalog_manager.cc:5654] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.720959 21173 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.721679 21699 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.722023 21173 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.715907 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.716219 21829 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:39.716797 21783 consensus_queue.cc:1059] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.722841 21275 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:39.717442 21790 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:39.724001 21829 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.703620 21793 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251214 02:58:39.722837 21699 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.723585 21783 raft_consensus.cc:3055] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.725785 21783 raft_consensus.cc:740] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.726024 21784 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.726822 21783 consensus_queue.cc:260] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.717506 21099 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.727905 21849 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.728089 21783 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.729851 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:39.724048 21173 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.735194 21793 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:39.735638 21793 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:39.735733 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.736049 21793 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.736496 21794 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.736864 21794 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:39.736835 21784 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:39.737389 21794 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.738629 21099 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:39.739424 21784 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.725848 21829 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.741642 21028 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 3, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.744969 21794 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.746739 21699 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.747169 21699 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:39.747529 21699 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.751003 21779 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 02:58:39.701483 21248 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:39.754014 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 52612.2us
I20251214 02:58:39.754717 21639 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.755172 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.754760 21248 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.756237 21804 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.756449 21027 catalog_manager.cc:5654] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:39.757525 21790 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.758440 21206 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.759106 21248 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.759778 21639 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.762117 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.763908 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.764557 21206 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:39.768888 21204 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251214 02:58:39.774853 21829 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.775709 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.776458 21694 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.777977 21173 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.778939 21849 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.779630 21694 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.780572 21794 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.787196 21699 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.789080 21851 consensus_queue.cc:1059] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.793706 21851 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.794168 21851 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.794373 21804 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.795378 21248 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.795164 21851 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.796469 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.796707 21851 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.797520 21248 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.798200 21786 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.798847 21786 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.799185 21327 tablet_bootstrap.cc:492] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:39.799383 21786 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.799665 21327 ts_tablet_manager.cc:1403] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.454s user 0.012s sys 0.009s
I20251214 02:58:39.800765 21248 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.801545 21804 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.802623 21248 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.804167 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.804884 21639 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:39.805227 21639 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:39.805562 21639 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.815313 21639 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.816668 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.817720 21786 consensus_queue.cc:1048] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.818161 21787 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.819415 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.820163 21786 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:39.820567 21786 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:39.821126 21786 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.824476 21279 raft_consensus.cc:1240] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.826018 21779 consensus_queue.cc:1059] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.827171 21790 raft_consensus.cc:3055] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.827567 21790 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.828423 21790 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.829974 21804 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.820549 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.802449 21327 raft_consensus.cc:359] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.835896 21327 raft_consensus.cc:385] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:39.835983 21248 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.838704 21804 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.838963 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:39.840026 21639 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:39.836328 21327 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:39.792104 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.841439 21327 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.842247 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:58:39.840873 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 48177us
I20251214 02:58:39.843510 21248 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.844580 21699 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:39.845070 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 53058.5us
I20251214 02:58:39.853338 21794 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.857872 21790 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.846012 21804 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.864217 21793 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.866120 21793 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.853451 21787 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.868261 21327 ts_tablet_manager.cc:1434] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.068s user 0.004s sys 0.002s
I20251214 02:58:39.869333 21327 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Bootstrap starting.
I20251214 02:58:39.876492 21783 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:39.876709 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:39.879365 21787 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.853660 21639 consensus_queue.cc:1059] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251214 02:58:39.882287 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.880818 21639 raft_consensus.cc:3055] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.882884 21202 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.883013 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.880950 21205 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.889715 21202 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:39.891088 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:39.891908 21783 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:39.892379 21783 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.892633 21208 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:39.893168 21783 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.895026 21173 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.883002 21639 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.895697 21786 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.896842 21276 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 1: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:39.896538 21639 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.898164 21639 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.900779 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:39.897588 21786 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:39.908766 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.918257 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.921392 21804 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:39.922001 21804 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.922956 21804 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.925129 21027 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.927345 21027 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.933466 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:39.941010 21201 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 1: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:39.932446 21784 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:39.942962 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.942183 21784 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.944047 21811 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.936506 21808 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.926931 21794 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:39.929404 21786 consensus_queue.cc:1048] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251214 02:58:39.940155 21278 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251214 02:58:39.930720 21829 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.934522 21199 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.945087 21027 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.947293 21784 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:39.948033 21027 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.949240 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:39.949873 21199 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:39.950939 21690 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.951289 21690 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:39.952060 21690 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.953384 21099 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:39.954375 21784 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:39.954954 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:39.955181 21845 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 02:58:39.955518 21845 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:39.955585 21278 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 0.
I20251214 02:58:39.949735 21027 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:39.955878 21845 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:39.957662 21851 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251214 02:58:39.955773 21784 raft_consensus.cc:3055] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.963474 21690 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.966274 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.968654 21199 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:39.966060 21784 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:39.978797 21644 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.981628 21327 tablet_bootstrap.cc:654] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Neither blocks nor log segments found. Creating new log.
I20251214 02:58:39.979475 21173 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.983636 21173 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:39.985388 21644 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.985524 21031 catalog_manager.cc:5654] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:39.984485 21794 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:39.981844 21784 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:39.989066 21784 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:39.982100 21783 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251214 02:58:40.000299 21849 raft_consensus.cc:3055] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:39.990476 21173 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.002807 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.001804 21849 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:40.004530 21644 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 02:58:40.004925 21849 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.006207 21849 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.007064 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.008217 21794 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.010479 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.011962 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.015588 21699 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:40.003868 21829 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.015347 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.004691 21845 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.016029 21699 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:40.015262 21644 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.015585 21867 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:40.018781 21867 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.017405 21173 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.017066 21694 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.021390 21867 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:40.021767 21699 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.030428 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.029373 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:40.035226 21694 raft_consensus.cc:2764] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251214 02:58:40.035805 21199 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:40.036538 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.037148 21276 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:40.035414 21867 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.042662 21845 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:40.044653 21199 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:40.044587 21276 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:40.046159 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.046228 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.047360 21867 raft_consensus.cc:2764] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251214 02:58:40.047926 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.048959 21867 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.047523 21784 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:40.064103 21784 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:40.065004 21784 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.050248 21694 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:40.051333 21829 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.052412 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.069185 21829 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.069286 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.070533 21829 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.072633 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:40.071990 21867 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.073791 21327 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: No bootstrap required, opened a new log
I20251214 02:58:40.074299 21327 ts_tablet_manager.cc:1403] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Time spent bootstrapping tablet: real 0.205s user 0.012s sys 0.005s
I20251214 02:58:40.076138 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:40.076972 21199 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.077049 21327 raft_consensus.cc:359] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.077795 21327 raft_consensus.cc:385] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 02:58:40.078267 21327 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Initialized, Role: FOLLOWER
I20251214 02:58:40.078706 21127 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.078963 21327 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.080004 21845 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.081274 21327 ts_tablet_manager.cc:1434] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20251214 02:58:40.083184 21127 raft_consensus.cc:1240] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.086118 21276 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.086740 21699 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.087692 21849 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.100430 21126 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.074488 21867 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.104770 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:40.106077 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:40.106767 21279 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:40.109659 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.111109 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.114120 21280 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251214 02:58:40.114800 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:40.115098 21125 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.115549 21129 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:40.115675 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.108870 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:40.120568 21204 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.114364 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:40.127271 21690 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:40.133064 21699 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:40.132862 21690 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.134858 21690 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:40.135771 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:40.137638 21694 consensus_queue.cc:1059] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.140000 21794 raft_consensus.cc:3055] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.140370 21794 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.140688 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:40.141957 21247 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.141099 21794 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.147308 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.147514 21794 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.129701 21031 catalog_manager.cc:5654] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:40.108160 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.149298 21031 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:40.152325 21133 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:40.156229 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.159122 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.159722 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.160183 21281 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.161697 21130 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.164215 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:40.164832 21124 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 0.
I20251214 02:58:40.148244 21690 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.169171 21644 consensus_queue.cc:1059] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.170029 21829 raft_consensus.cc:3055] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.170145 21783 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:40.170380 21829 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.171201 21829 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.172224 21829 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.183921 21767 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:40.155288 21278 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:40.189945 21131 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
W20251214 02:58:40.210830 21872 log.cc:927] Time spent T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20251214 02:58:40.216008 21028 catalog_manager.cc:5654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:58:40.225767 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 43736.2us
I20251214 02:58:40.185789 21644 consensus_queue.cc:1059] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251214 02:58:40.227362 21281 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.235838 21128 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:40.240747 21790 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.241568 21644 raft_consensus.cc:3055] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.241937 21644 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.234174 21694 consensus_queue.cc:1059] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251214 02:58:40.243505 21694 raft_consensus.cc:3055] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.243824 21694 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.244537 21694 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.256201 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.233520 21126 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:40.256171 21828 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251214 02:58:40.268055 21282 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.268527 21204 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.269891 21848 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.261415 21694 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.254477 21644 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.265334 21878 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.307960 21783 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.285236 21828 consensus_queue.cc:1243] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 721d604130924d939d291fe3eef8f6d1 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251214 02:58:40.292236 21878 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.334921 21204 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.341709 21878 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.354285 21849 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 02:58:40.377722 21204 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:58:40.379532 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 123354us
W20251214 02:58:40.378803 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:40.380651 21783 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:40.392746 21173 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.404259 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.404462 21699 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251214 02:58:40.468063 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63879.8us
W20251214 02:58:40.410988 21881 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20251214 02:58:40.404042 21828 consensus_queue.cc:1059] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20251214 02:58:40.468884 21173 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:58:40.537045 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 68266.7us
I20251214 02:58:40.537710 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.495996 21208 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.518479 21644 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.542244 21867 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.542662 21867 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.542950 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.543819 21126 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:40.543474 21867 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.469269 21885 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.544727 21867 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.495371 21783 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20251214 02:58:40.547855 21879 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.548389 21879 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.549312 21879 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.550765 21885 raft_consensus.cc:697] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.551757 21885 consensus_queue.cc:237] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.552039 21787 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:58:40.560729 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:40.552887 21694 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.561695 21694 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.562533 21694 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.509505 21829 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.563879 21829 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.564673 21829 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.530833 21201 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:58:40.588889 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.468870 21699 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:40.591693 21201 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.606626 21131 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:58:40.606858 21201 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.608003 21201 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.594547 21699 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 0 FOLLOWER]: Advancing to term 1
W20251214 02:58:40.662245 21079 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251214 02:58:40.651957 21829 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.666071 21279 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:40.719254 21644 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:58:40.719934 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:40.719842 21644 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.721679 21644 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:58:40.715142 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:40.722702 21173 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.714849 21128 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:58:40.736737 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 52437.1us
I20251214 02:58:40.737572 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.732874 21828 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.765664 21173 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.770159 21895 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.776809 21201 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:58:40.777455 21794 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20251214 02:58:40.779023 21892 log.cc:927] Time spent T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20251214 02:58:40.791946 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49269.9us
I20251214 02:58:40.807186 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:40.808146 21694 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:40.808550 21694 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:40.808898 21694 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.814028 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:40.815065 21644 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:40.815498 21644 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:40.816236 21644 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.821317 21273 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.795020 21893 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:58:40.828619 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:40.829293 21126 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
W20251214 02:58:40.829787 21896 log.cc:927] Time spent T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20251214 02:58:40.830367 21786 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.836416 21694 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.845785 21699 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:40.846892 21851 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 02:58:40.847764 21699 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:40.838375 21694 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:40.849671 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:58:40.850327 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:58:40.851802 21282 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:40.860695 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:40.861717 21794 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:40.862103 21794 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:40.862417 21794 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:58:40.905586 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 159239us
I20251214 02:58:40.908484 21128 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:40.900554 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:40.909013 21793 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:40.883315 21885 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251214 02:58:40.904178 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:40.919129 21130 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:40.912498 21894 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20251214 02:58:40.914173 21791 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20251214 02:58:40.912490 21281 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 0 FOLLOWER]: Advancing to term 1
I20251214 02:58:40.947250 21787 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:40.927064 21273 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:40.949898 21787 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.007731 21787 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.007861 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.008666 21124 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:41.029740 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.030450 21199 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:41.030920 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.031527 21124 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:58:41.006081 21281 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:41.033004 21248 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:41.035434 21173 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:41.006074 21794 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.046530 21131 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.046531 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:58:41.048969 21907 log.cc:927] Time spent T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20251214 02:58:41.033691 21787 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.051272 21893 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 02:58:41.052445 21893 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:41.053247 21893 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.041361 21791 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.051744 21787 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.062740 21128 raft_consensus.cc:1275] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.067425 21130 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:41.061113 21787 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.074313 21027 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.075955 21027 catalog_manager.cc:5654] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.077734 21787 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.077525 21027 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.079305 21273 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.080092 21124 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.061345 21867 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251214 02:58:41.084128 21848 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:41.084352 21794 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.084631 21848 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.087563 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.088188 21273 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
W20251214 02:58:41.088981 21904 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20251214 02:58:41.089632 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.090873 21849 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:41.061635 21791 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.093413 21791 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.091203 21849 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:41.099422 21027 catalog_manager.cc:5654] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.100682 21878 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.102435 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.102895 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.103449 21128 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.103031 21200 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.105120 21276 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:41.094231 21849 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.110415 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.086465 21848 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.111482 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:41.113060 21644 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:41.113476 21644 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:41.117457 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.118245 21203 raft_consensus.cc:2393] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:41.111845 21879 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251214 02:58:41.122143 21128 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:58:41.127395 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.128823 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.128932 21804 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.129827 21804 raft_consensus.cc:697] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:58:41.131271 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.126494 21282 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.136466 21273 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.138109 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:41.141194 21845 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:58:41.141139 21644 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.138757 21804 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.108176 21787 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.147312 21878 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.122417 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.151084 21281 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.152032 21282 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.155296 21279 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.162685 21279 raft_consensus.cc:1275] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.166031 21130 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.168697 21794 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251214 02:58:41.171200 21878 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251214 02:58:41.204950 21128 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.210745 21644 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:41.209975 21273 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.211866 21200 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:58:41.260937 21878 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.261662 21130 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.272044 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.272782 21878 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.273340 21878 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:41.274248 21878 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.277690 21281 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.278174 21848 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.231104 21849 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.278672 21848 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.280258 21849 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.280460 21848 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.250025 21783 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.281234 21783 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.281577 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.282246 21207 raft_consensus.cc:2393] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:41.283072 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.283687 21203 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.284633 21644 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.285053 21644 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.286614 21099 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.287631 21778 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.287770 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.288215 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.288785 21207 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.290369 21644 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.290591 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.283083 21783 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.291355 21099 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.292526 21783 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.292862 21783 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.293187 21783 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.293239 21778 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.294018 21778 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.294368 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.296507 21791 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.296933 21791 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.299072 21791 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.309770 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.314742 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.315608 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.317725 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.322988 21280 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:58:41.324958 21277 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.318291 21132 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.321485 21124 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:41.333230 21098 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:41.335655 21900 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.340349 21900 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.340855 21900 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.358888 21829 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.359179 21778 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.360760 21173 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.363420 21829 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.363751 21829 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.364042 21829 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.369472 21281 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.371282 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:41.371146 21829 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.361567 21778 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.361819 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.388005 21207 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.389583 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:58:41.430485 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57600.3us
I20251214 02:58:41.434298 21878 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.434913 21878 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:58:41.435217 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57501.8us
I20251214 02:58:41.440054 21207 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.443280 21099 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.443991 21905 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.444439 21905 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.434316 21829 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.445299 21905 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.436892 21878 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.455487 21783 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.457383 21783 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.458259 21791 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:41.458860 21791 raft_consensus.cc:697] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.459846 21791 consensus_queue.cc:237] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.462468 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.462901 21902 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:41.453394 21900 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.463081 21207 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.463464 21902 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.465375 21902 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.465332 21778 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.466264 21778 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.468024 21778 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.471609 21207 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.474184 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.475719 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.475173 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.475857 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.476318 21202 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.476469 21132 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:41.476016 21200 raft_consensus.cc:2393] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:41.474737 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.478039 21124 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.478485 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.479039 21273 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.479694 21900 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.481777 21099 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.486838 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.487473 21277 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.488018 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.488692 21276 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.490404 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.491019 21275 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.486372 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.499838 21280 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:41.499722 21173 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:41.502290 21878 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.503011 21878 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.503695 21878 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.505299 21273 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.510627 21905 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.513252 21275 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.515350 21099 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.519621 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.519145 21028 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:58:41.519125 21848 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.523734 21644 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.529438 21173 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:41.531304 21644 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.531796 21644 raft_consensus.cc:697] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:41.532555 21644 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.520747 21848 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.547298 21848 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.544273 21124 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.560240 21202 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.535033 21905 raft_consensus.cc:697] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.556309 21791 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.583331 21098 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:41.585425 21132 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:58:41.571880 21879 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:41.586421 21879 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.587839 21879 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.587111 21029 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.590389 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.591102 21282 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:41.589233 21029 catalog_manager.cc:5654] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.570817 21690 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20251214 02:58:41.561686 21878 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.598322 21029 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:58:41.606825 21778 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:41.607265 21778 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:58:41.607610 21778 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.603871 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.598415 21893 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251214 02:58:41.614697 21893 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:41.615051 21893 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:41.616146 21787 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:41.616659 21787 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.597462 21894 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20251214 02:58:41.615826 21893 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.623675 21098 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:41.577842 21791 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.624123 21690 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.634783 21690 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.636727 21690 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.638998 21804 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:41.639532 21804 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.641415 21804 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.630043 21902 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.644126 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.623996 21894 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20251214 02:58:41.644989 21201 raft_consensus.cc:2368] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:41.620038 21787 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.620263 21893 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.649935 21248 leader_election.cc:400] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:41.651012 21248 leader_election.cc:403] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:58:41.652338 21902 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.654538 21902 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.656212 21791 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.657815 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.658418 21201 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:58:41.658821 21132 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:41.661839 21778 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.629487 21905 consensus_queue.cc:237] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.663718 21778 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.665050 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.665733 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.666899 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.668807 21133 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.681785 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.670778 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.687978 21130 raft_consensus.cc:2393] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:41.682550 21202 raft_consensus.cc:2393] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:58:41.688874 21878 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.689188 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.689829 21199 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:58:41.691212 21248 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:58:41.691332 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.691747 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.691885 21130 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.692900 21173 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:41.666430 21787 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.694883 21794 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:58:41.695290 21794 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:58:41.695637 21794 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:58:41.699687 21130 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.681282 21203 raft_consensus.cc:2368] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:41.682870 21778 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.705639 21690 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.710805 21690 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.691862 21848 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.710323 21778 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.711253 21690 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.717614 21028 catalog_manager.cc:5654] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.723471 21778 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.724942 21804 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.725128 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.736769 21126 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.737324 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.737859 21279 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.738096 21694 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.738631 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.739203 21203 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.746325 21935 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.748107 21279 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.749425 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.755971 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.757187 21173 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:58:41.756210 21028 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.760970 21893 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.761682 21893 raft_consensus.cc:697] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:58:41.763975 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 72881.1us
I20251214 02:58:41.746804 21935 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.765273 21028 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.733702 21848 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.769035 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.762620 21893 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.772008 21248 leader_election.cc:400] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:41.759788 21934 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.773885 21276 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.821494 21794 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.748862 21280 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.786260 21203 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.787448 21935 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.789559 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:41.791862 21787 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:41.760322 21778 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.800370 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.809651 21690 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.775619 21248 leader_election.cc:403] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:58:41.829188 21276 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:41.830922 21098 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:58:41.834084 21128 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:41.836087 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.836503 21278 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:41.837224 21787 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
W20251214 02:58:41.837427 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 65561.1us
I20251214 02:58:41.841192 21848 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:58:41.842200 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.842803 21199 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.832561 21934 raft_consensus.cc:697] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.848878 21276 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.846088 21848 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.853878 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.834872 21202 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:58:41.854701 21276 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:58:41.857657 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.854113 21778 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.860293 21099 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:41.862792 21783 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.863722 21783 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.863437 21934 consensus_queue.cc:237] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.864086 21783 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.865799 21202 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.867378 21935 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.855576 21794 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.874815 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:41.878062 21276 raft_consensus.cc:2368] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:58:41.857082 21690 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:41.860312 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:41.880074 21132 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:58:41.853861 21848 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.881369 21247 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:41.882057 21690 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:58:41.888104 21173 leader_election.cc:400] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:58:41.888419 21173 leader_election.cc:403] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251214 02:58:41.887459 21829 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20251214 02:58:41.896255 21276 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.902380 21199 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:41.903235 21694 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:58:41.903460 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:41.903712 21694 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.904466 21778 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:41.904927 21778 raft_consensus.cc:697] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:41.905565 21694 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:41.905696 21778 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.907469 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:41.907985 21276 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.907681 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:58:41.921891 21029 catalog_manager.cc:5654] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.936744 21029 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:41.908732 21794 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:58:41.919199 21275 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:58:41.919001 21784 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251214 02:58:41.953292 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:41.954059 21794 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:58:41.954420 21794 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:58:41.954758 21794 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:41.983284 21202 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:41.988104 21900 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:41.989718 21277 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:41.950454 21783 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:41.992136 21783 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:42.000644 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:58:41.997601 21028 catalog_manager.cc:5654] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.002454 21028 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.003628 21028 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.001507 21202 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:42.001511 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:42.012706 21848 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251214 02:58:42.062834 21277 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.076397 21274 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.080713 21814 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.082047 21885 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.080601 21794 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:42.103046 21794 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:42.103435 21202 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:58:42.111660 21786 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:58:42.112677 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:58:42.113250 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:58:42.120334 21274 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:42.116415 21786 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:42.124799 21786 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:42.128108 21099 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:58:42.118238 21200 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.132793 21934 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:42.136049 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:58:42.141382 21202 raft_consensus.cc:2368] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:42.142527 21248 leader_election.cc:400] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:42.142833 21248 leader_election.cc:403] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:58:42.143728 21786 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:58:42.144685 21126 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.139045 21934 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:42.146087 21885 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.145931 21934 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:42.138147 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:58:42.164445 21027 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.166216 21027 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.177682 21934 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.195667 21203 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251214 02:58:42.196062 21203 pending_rounds.cc:85] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 1
I20251214 02:58:42.196291 21203 pending_rounds.cc:107] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251214 02:58:42.196532 21203 raft_consensus.cc:2889] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:58:42.228173 21274 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:58:42.229786 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:58:42.230417 21786 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:58:42.230841 21885 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:58:42.231330 21885 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:58:42.233371 21885 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:42.235406 21784 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.241789 21277 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.242771 21029 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.244457 21848 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.320258 21029 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:42.346154 21203 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:58:42.347448 21848 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.348177 21848 consensus_queue.cc:1243] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer 668ad4f30bd649c88a034bbab9f13b20 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251214 02:58:42.348548 21784 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:58:42.348888 21784 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:58:42.349594 21784 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:58:42.350723 21784 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:58:42.354843 21203 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.356240 21848 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.374161 21274 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.377750 21935 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.403646 21275 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.405021 21848 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.440541 21274 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.443254 21893 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:58:42.465781 21965 log.cc:927] Time spent T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20251214 02:58:42.480607 21199 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.484472 21935 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.499254 21132 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.500733 21794 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.630473 21132 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251214 02:58:42.630858 21132 pending_rounds.cc:85] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 1
I20251214 02:58:42.631054 21132 pending_rounds.cc:107] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251214 02:58:42.631249 21132 raft_consensus.cc:2889] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:58:42.632459 21794 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.652976 21274 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.661252 21829 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.666553 19981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 02:58:42.679032 21203 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.679837 21274 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:58:42.680771 21848 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:42.681541 21956 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:58:42.684688 21971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:42.717370 21972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 02:58:42.763729 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 02:58:42.767004 19981 server_base.cc:1047] running on GCE node
I20251214 02:58:42.768348 19981 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251214 02:58:42.768669 19981 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251214 02:58:42.768880 19981 hybrid_clock.cc:648] HybridClock initialized: now 1765681122768862 us; error 0 us; skew 500 ppm
I20251214 02:58:42.769584 19981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 02:58:42.810417 19981 webserver.cc:492] Webserver started at http://127.19.131.68:43819/ using document root <none> and password file <none>
I20251214 02:58:42.811702 19981 fs_manager.cc:362] Metadata directory not provided
I20251214 02:58:42.812074 19981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 02:58:42.812520 19981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 02:58:42.814254 19981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-3-root/instance:
uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
format_stamp: "Formatted at 2025-12-14 02:58:42 on dist-test-slave-8pjb"
I20251214 02:58:42.822700 19981 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20251214 02:58:42.835489 21981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:42.836735 19981 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20251214 02:58:42.837217 19981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-3-root
uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
format_stamp: "Formatted at 2025-12-14 02:58:42 on dist-test-slave-8pjb"
I20251214 02:58:42.837693 19981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 02:58:42.912724 19981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 02:58:42.914551 19981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 02:58:42.922529 19981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 02:58:42.925956 19981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 02:58:42.926219 19981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 02:58:42.926512 19981 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 02:58:42.926738 19981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251214 02:58:43.548444 19981 rpc_server.cc:307] RPC server started. Bound to: 127.19.131.68:33395
I20251214 02:58:43.552757 22044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.131.68:33395 every 8 connection(s)
I20251214 02:58:43.566588 19981 auto_leader_rebalancer-test.cc:417] add a tserver: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:43.608250 22045 heartbeater.cc:344] Connected to a master server at 127.19.131.126:41013
I20251214 02:58:43.608757 22045 heartbeater.cc:461] Registering TS with master...
I20251214 02:58:43.609613 22045 heartbeater.cc:507] Master 127.19.131.126:41013 requested a full tablet report, sending...
I20251214 02:58:43.611761 21027 ts_manager.cc:194] Registered new tserver with Master: 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:58:43.613548 21027 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59016
I20251214 02:58:44.508193 21274 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:44.543289 21132 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:44.550462 21200 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:44.552695 21942 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:44.554538 21942 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:44.599750 21982 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.604238 21126 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.606058 21200 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.616803 22045 heartbeater.cc:499] Master 127.19.131.126:41013 was elected leader, sending a full tablet report...
W20251214 02:58:44.645083 21245 consensus_peers.cc:597] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8ab1ed25694443cb4efebfc35e5205e. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:44.644444 21029 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.706278 21126 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:44.742507 21276 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:44.743217 21200 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:44.744052 21934 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:44.749675 22054 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:44.777487 21845 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
W20251214 02:58:44.779806 21096 consensus_peers.cc:597] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2953f784a3f34b61a952ba296007ae31. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:44.779107 21276 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.797959 21027 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.811041 21200 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.872545 21200 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:44.885270 21276 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:44.886857 22053 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:44.899428 21126 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:44.903393 21894 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:58:44.906909 21171 consensus_peers.cc:597] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5a50d11dd114861bc3ebe1138e3d999. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:44.922302 22053 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.930415 21126 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:44.960050 22074 ts_tablet_manager.cc:933] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:44.972481 22074 tablet_copy_client.cc:323] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:58:44.982677 21200 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:44.993891 21274 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:45.025125 21027 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:45.046238 21292 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet c8ab1ed25694443cb4efebfc35e5205e from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:58:45.046897 21292 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet c8ab1ed25694443cb4efebfc35e5205e from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-c8ab1ed25694443cb4efebfc35e5205e
I20251214 02:58:45.063426 21292 tablet_copy_source_session.cc:215] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:45.070626 22074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ab1ed25694443cb4efebfc35e5205e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 02:58:45.090412 21171 consensus_peers.cc:597] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 278f5a62c6a74101ae76cfc7325bcca0. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:45.093986 22074 tablet_copy_client.cc:806] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:45.094107 21276 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:45.095732 22074 tablet_copy_client.cc:670] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:45.095785 21126 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:45.107350 21794 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:45.115072 22053 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:45.129364 22074 tablet_copy_client.cc:538] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:45.149212 22074 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:45.151422 21894 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:45.153550 21126 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:45.166499 21276 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:45.188412 22081 ts_tablet_manager.cc:933] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:45.187947 21027 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:45.213511 22074 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:45.214582 22074 tablet_bootstrap.cc:492] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:45.215610 22074 ts_tablet_manager.cc:1403] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.012s
I20251214 02:58:45.216699 22081 tablet_copy_client.cc:323] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:45.226538 22074 raft_consensus.cc:359] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.229338 22074 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:45.230512 22074 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.250248 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 2953f784a3f34b61a952ba296007ae31 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:45.250897 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 2953f784a3f34b61a952ba296007ae31 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-2953f784a3f34b61a952ba296007ae31
I20251214 02:58:45.271665 21143 tablet_copy_source_session.cc:215] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:45.290409 22081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2953f784a3f34b61a952ba296007ae31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:45.316370 22081 tablet_copy_client.cc:806] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:45.319181 22081 tablet_copy_client.cc:670] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:45.321362 22074 ts_tablet_manager.cc:1434] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.104s user 0.008s sys 0.004s
I20251214 02:58:45.323321 21292 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-c8ab1ed25694443cb4efebfc35e5205e received from {username='slave'} at 127.0.0.1:50140
I20251214 02:58:45.323794 21292 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-c8ab1ed25694443cb4efebfc35e5205e on tablet c8ab1ed25694443cb4efebfc35e5205e with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:45.356969 22081 tablet_copy_client.cc:538] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:45.369813 22020 raft_consensus.cc:1217] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:58:45.402076 22081 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:45.447777 22081 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:45.448843 22081 tablet_bootstrap.cc:492] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:45.449596 22081 ts_tablet_manager.cc:1403] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.048s user 0.029s sys 0.004s
I20251214 02:58:45.452198 22081 raft_consensus.cc:359] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.453083 22081 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:45.455828 22081 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.459985 22081 ts_tablet_manager.cc:1434] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251214 02:58:45.461949 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-2953f784a3f34b61a952ba296007ae31 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:45.462363 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-2953f784a3f34b61a952ba296007ae31 on tablet 2953f784a3f34b61a952ba296007ae31 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:45.491847 22081 ts_tablet_manager.cc:933] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:45.503695 22081 tablet_copy_client.cc:323] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:45.550963 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet d5a50d11dd114861bc3ebe1138e3d999 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:45.551631 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet d5a50d11dd114861bc3ebe1138e3d999 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-d5a50d11dd114861bc3ebe1138e3d999
I20251214 02:58:45.568713 21218 tablet_copy_source_session.cc:215] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:45.572295 22081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a50d11dd114861bc3ebe1138e3d999. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:45.654121 22081 tablet_copy_client.cc:806] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:45.654693 22081 tablet_copy_client.cc:670] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:45.663981 22081 tablet_copy_client.cc:538] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:45.673520 22074 ts_tablet_manager.cc:933] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:45.674242 22081 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:45.708793 22074 tablet_copy_client.cc:323] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:45.710512 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 278f5a62c6a74101ae76cfc7325bcca0 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:45.711045 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 278f5a62c6a74101ae76cfc7325bcca0 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-278f5a62c6a74101ae76cfc7325bcca0
I20251214 02:58:45.717379 21218 tablet_copy_source_session.cc:215] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:45.720249 22081 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:45.720860 22074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f5a62c6a74101ae76cfc7325bcca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:45.721123 22081 tablet_bootstrap.cc:492] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:45.721807 22081 ts_tablet_manager.cc:1403] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.000s
I20251214 02:58:45.724946 22081 raft_consensus.cc:359] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.725708 22081 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:45.726787 22081 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.740904 22081 ts_tablet_manager.cc:1434] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20251214 02:58:45.743211 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-d5a50d11dd114861bc3ebe1138e3d999 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:45.743705 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-d5a50d11dd114861bc3ebe1138e3d999 on tablet d5a50d11dd114861bc3ebe1138e3d999 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:45.759917 22020 raft_consensus.cc:1217] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:58:45.762049 22074 tablet_copy_client.cc:806] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:45.762683 22074 tablet_copy_client.cc:670] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:45.768607 22074 tablet_copy_client.cc:538] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:45.777333 22074 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:45.810274 22074 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:45.811331 22074 tablet_bootstrap.cc:492] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:45.812067 22074 ts_tablet_manager.cc:1403] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.014s
I20251214 02:58:45.814649 22074 raft_consensus.cc:359] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.815517 22074 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:45.816439 22074 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:45.825090 22074 ts_tablet_manager.cc:1434] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251214 02:58:45.833165 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-278f5a62c6a74101ae76cfc7325bcca0 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:45.833604 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-278f5a62c6a74101ae76cfc7325bcca0 on tablet 278f5a62c6a74101ae76cfc7325bcca0 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:45.883059 22086 raft_consensus.cc:1064] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:45.885653 22086 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:45.903683 22020 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:45.904868 21207 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:45.905164 22085 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:45.907352 22085 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:45.950848 21132 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:45.962476 22086 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251214 02:58:45.989234 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 83098us
I20251214 02:58:46.010841 22085 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.021013 21132 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.043849 21027 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.074915 22020 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.102316 22019 raft_consensus.cc:1217] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:46.141804 21199 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.174496 21275 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.190296 21126 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:46.190506 22020 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:46.195040 22095 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:46.198578 22095 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:46.221271 22097 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.223977 21132 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.237156 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet c8ab1ed25694443cb4efebfc35e5205e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:46.250538 21029 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.259855 22019 raft_consensus.cc:2955] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.254900 22020 raft_consensus.cc:1217] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:46.303285 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet c8ab1ed25694443cb4efebfc35e5205e with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:46.324111 22104 tablet_replica.cc:333] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:46.357196 22090 raft_consensus.cc:1064] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:46.361541 22104 raft_consensus.cc:2243] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:46.362313 22104 raft_consensus.cc:2272] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:46.361862 22090 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.366348 22104 ts_tablet_manager.cc:1916] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:46.375290 22019 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:46.376848 21275 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:46.376958 22054 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:46.378782 21207 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:46.381183 21934 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:46.382601 22054 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:46.399439 22104 ts_tablet_manager.cc:1929] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:58:46.400030 22104 log.cc:1199] T c8ab1ed25694443cb4efebfc35e5205e P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/c8ab1ed25694443cb4efebfc35e5205e
I20251214 02:58:46.401714 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet c8ab1ed25694443cb4efebfc35e5205e (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:46.424818 21784 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.428758 21202 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.446156 22019 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.456799 21029 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:58:46.476855 22107 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20251214 02:58:46.492409 21277 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:58:46.522114 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 45154.8us
I20251214 02:58:46.525471 21132 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.535326 21199 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:46.536175 22019 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:46.536918 21784 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:46.548431 22090 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:46.554831 21934 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.566992 21202 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.587591 22019 raft_consensus.cc:2955] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.590314 21029 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.601428 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 2953f784a3f34b61a952ba296007ae31 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50512
I20251214 02:58:46.616147 22110 tablet_replica.cc:333] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:58:46.617272 22110 raft_consensus.cc:2243] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:46.618943 22110 raft_consensus.cc:2272] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:46.622236 22102 raft_consensus.cc:1064] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:46.624837 22102 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.627781 22110 ts_tablet_manager.cc:1916] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:46.643960 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 2953f784a3f34b61a952ba296007ae31 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:46.667836 22110 ts_tablet_manager.cc:1929] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:58:46.668381 22110 log.cc:1199] T 2953f784a3f34b61a952ba296007ae31 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/2953f784a3f34b61a952ba296007ae31
I20251214 02:58:46.669999 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 2953f784a3f34b61a952ba296007ae31 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:46.703650 22019 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.713554 22118 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:46.776692 21132 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.782361 21275 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.793773 22102 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:46.800835 22102 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:46.833173 22102 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.851293 21132 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.853480 21275 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.857383 22019 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.862996 22111 raft_consensus.cc:1064] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:46.865422 22111 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.869629 21031 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:46.902737 21208 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:46.914800 22019 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.916503 21132 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.916471 22102 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:46.918195 22118 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:46.933200 21275 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:46.935891 22111 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:46.946789 21277 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:46.950093 22117 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:46.952855 22053 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.960274 21277 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:46.991134 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d5a50d11dd114861bc3ebe1138e3d999 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:46.994158 22020 raft_consensus.cc:2955] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.001822 21031 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.017046 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet d5a50d11dd114861bc3ebe1138e3d999 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47052
I20251214 02:58:47.041973 22127 tablet_replica.cc:333] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:58:47.061057 22127 raft_consensus.cc:2243] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:47.062736 22127 raft_consensus.cc:2272] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:47.064889 22111 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.065857 21132 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.072945 22127 ts_tablet_manager.cc:1916] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:47.090044 22127 ts_tablet_manager.cc:1929] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:47.090840 21275 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.094507 22020 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.109602 21031 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.090854 22127 log.cc:1199] T d5a50d11dd114861bc3ebe1138e3d999 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/d5a50d11dd114861bc3ebe1138e3d999
I20251214 02:58:47.114653 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet d5a50d11dd114861bc3ebe1138e3d999 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:47.147106 21204 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:47.152765 22020 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:47.152998 21275 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:47.154229 22117 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:47.155120 22111 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:47.183600 22118 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.193243 22019 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.199445 21276 raft_consensus.cc:2955] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.210063 21031 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:47.214155 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 278f5a62c6a74101ae76cfc7325bcca0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:47.221746 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet 278f5a62c6a74101ae76cfc7325bcca0 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47052
I20251214 02:58:47.222692 22127 tablet_replica.cc:333] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:58:47.223636 22127 raft_consensus.cc:2243] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:47.224225 22127 raft_consensus.cc:2272] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:47.228085 22127 ts_tablet_manager.cc:1916] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:47.263078 22127 ts_tablet_manager.cc:1929] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:47.263520 22127 log.cc:1199] T 278f5a62c6a74101ae76cfc7325bcca0 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/278f5a62c6a74101ae76cfc7325bcca0
I20251214 02:58:47.264887 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet 278f5a62c6a74101ae76cfc7325bcca0 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:48.360612 21199 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
W20251214 02:58:48.377612 21171 consensus_peers.cc:597] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 172b9f2bc43b42fca62c756aa5cf17a1. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:48.379608 21277 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:48.381222 22118 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:48.387535 21126 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:48.389034 22118 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:48.436687 22118 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.438452 21132 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.446146 21275 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.462149 21126 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:48.466329 21031 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.485426 21275 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251214 02:58:48.486150 21096 consensus_peers.cc:597] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75e354ca56944ff1b7775a8309bca568. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:48.488330 22054 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:48.489326 21207 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:48.491166 22054 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:48.500226 22109 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.501626 21277 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.525786 21126 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:48.562489 21207 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
W20251214 02:58:48.579159 21096 consensus_peers.cc:597] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b9c9a9914df4ca0aa72659e67ec2c4c. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:48.579381 21277 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:58:48.582166 21199 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:58:48.581125 22109 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251214 02:58:48.585225 22090 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:48.603997 21845 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.611263 21207 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.618197 21029 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:48.623262 21199 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:48.651361 21126 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:48.651361 21277 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:58:48.653441 21171 consensus_peers.cc:597] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b611d0c068c451bb4eeb7eda4823036. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:48.654258 22119 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:48.650033 21029 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:48.655498 22117 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:48.662928 21274 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.700635 22118 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.704774 21132 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.736707 21029 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.743856 21277 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:48.862447 22149 ts_tablet_manager.cc:933] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:48.881151 22149 tablet_copy_client.cc:323] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:48.901369 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 172b9f2bc43b42fca62c756aa5cf17a1 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:48.902019 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 172b9f2bc43b42fca62c756aa5cf17a1 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-172b9f2bc43b42fca62c756aa5cf17a1
I20251214 02:58:48.914016 21218 tablet_copy_source_session.cc:215] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:48.931079 22151 ts_tablet_manager.cc:933] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:48.944826 22149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 172b9f2bc43b42fca62c756aa5cf17a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:48.966495 22151 tablet_copy_client.cc:323] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:48.968901 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 75e354ca56944ff1b7775a8309bca568 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:48.969467 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 75e354ca56944ff1b7775a8309bca568 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-75e354ca56944ff1b7775a8309bca568
I20251214 02:58:48.993355 22149 tablet_copy_client.cc:806] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:48.994246 22149 tablet_copy_client.cc:670] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:48.998652 21143 tablet_copy_source_session.cc:215] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:49.026489 22151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e354ca56944ff1b7775a8309bca568. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:49.001708 22149 tablet_copy_client.cc:538] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:49.045971 22149 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:49.082854 22151 tablet_copy_client.cc:806] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:49.083560 22151 tablet_copy_client.cc:670] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:49.105322 22154 ts_tablet_manager.cc:933] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:49.107672 22151 tablet_copy_client.cc:538] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:49.108580 22154 tablet_copy_client.cc:323] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:49.110162 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:49.110689 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 2b9c9a9914df4ca0aa72659e67ec2c4c from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-2b9c9a9914df4ca0aa72659e67ec2c4c
I20251214 02:58:49.113253 22149 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:49.114151 22149 tablet_bootstrap.cc:492] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:49.114831 22149 ts_tablet_manager.cc:1403] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.010s
I20251214 02:58:49.118032 22149 raft_consensus.cc:359] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.119266 22151 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:49.119254 22149 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:49.121546 21143 tablet_copy_source_session.cc:215] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:49.121088 22149 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.130841 22154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:49.136251 22149 ts_tablet_manager.cc:1434] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.021s user 0.015s sys 0.002s
I20251214 02:58:49.140815 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-172b9f2bc43b42fca62c756aa5cf17a1 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:49.141309 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-172b9f2bc43b42fca62c756aa5cf17a1 on tablet 172b9f2bc43b42fca62c756aa5cf17a1 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:49.177379 22154 tablet_copy_client.cc:806] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:49.182055 22151 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:49.188205 22151 tablet_bootstrap.cc:492] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:49.188911 22151 ts_tablet_manager.cc:1403] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.070s user 0.031s sys 0.006s
I20251214 02:58:49.190902 22154 tablet_copy_client.cc:670] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:49.191723 22151 raft_consensus.cc:359] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.192637 22151 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:49.193348 22151 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.195863 22151 ts_tablet_manager.cc:1434] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251214 02:58:49.197479 22154 tablet_copy_client.cc:538] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:49.198279 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-75e354ca56944ff1b7775a8309bca568 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:49.198732 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-75e354ca56944ff1b7775a8309bca568 on tablet 75e354ca56944ff1b7775a8309bca568 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:49.216751 22154 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:49.275079 22154 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:49.276065 22154 tablet_bootstrap.cc:492] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:49.276813 22154 ts_tablet_manager.cc:1403] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.023s
I20251214 02:58:49.278301 22151 ts_tablet_manager.cc:933] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:49.296134 22151 tablet_copy_client.cc:323] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:49.297593 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 4b611d0c068c451bb4eeb7eda4823036 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:49.298127 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 4b611d0c068c451bb4eeb7eda4823036 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-4b611d0c068c451bb4eeb7eda4823036
I20251214 02:58:49.299441 22154 raft_consensus.cc:359] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.300287 22154 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:49.300940 22154 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.303469 22154 ts_tablet_manager.cc:1434] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251214 02:58:49.307922 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-2b9c9a9914df4ca0aa72659e67ec2c4c received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:49.308359 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-2b9c9a9914df4ca0aa72659e67ec2c4c on tablet 2b9c9a9914df4ca0aa72659e67ec2c4c with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:49.312583 21218 tablet_copy_source_session.cc:215] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:49.315622 22151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b611d0c068c451bb4eeb7eda4823036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:49.331765 22151 tablet_copy_client.cc:806] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:49.333726 22151 tablet_copy_client.cc:670] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:49.342947 22151 tablet_copy_client.cc:538] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:49.352953 22151 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:49.374375 22151 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:49.375423 22151 tablet_bootstrap.cc:492] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:49.376163 22151 ts_tablet_manager.cc:1403] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20251214 02:58:49.379005 22151 raft_consensus.cc:359] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.379909 22151 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:49.380594 22151 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:49.388612 22151 ts_tablet_manager.cc:1434] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20251214 02:58:49.390305 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-4b611d0c068c451bb4eeb7eda4823036 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:49.390760 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-4b611d0c068c451bb4eeb7eda4823036 on tablet 4b611d0c068c451bb4eeb7eda4823036 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:49.447214 22019 raft_consensus.cc:1217] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:49.523408 22019 raft_consensus.cc:1217] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:49.538550 22019 raft_consensus.cc:1217] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251214 02:58:49.850957 22019 raft_consensus.cc:1217] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:58:49.901413 22111 raft_consensus.cc:1064] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:49.903640 22111 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:49.909839 21274 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:49.910073 21126 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:49.910073 22019 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:49.911604 22119 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:49.912299 22118 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:49.913013 22111 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:49.936157 22165 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.938337 21274 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.941535 22019 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.948870 21132 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.954708 22109 raft_consensus.cc:1064] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:49.956336 22109 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:49.969769 22020 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:58:49.969332 21274 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:58:49.971819 22090 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:49.962431 21027 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:49.975175 21845 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:49.992094 21274 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.990885 22090 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:49.993753 21208 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:58:49.998237 21845 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.020922 21208 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.029402 22020 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.032495 21207 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.043301 22019 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:50.044795 22111 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.045502 21126 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:50.048197 22118 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.051208 21027 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:50.052093 22165 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.054153 22018 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.063853 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 172b9f2bc43b42fca62c756aa5cf17a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:50.076093 21029 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.078372 21126 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.085089 21277 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:58:50.087123 22109 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:50.094404 22090 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.096282 21274 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.105387 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 172b9f2bc43b42fca62c756aa5cf17a1 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50512
I20251214 02:58:50.112354 22179 tablet_replica.cc:333] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:58:50.116575 22018 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:58:50.120554 22179 raft_consensus.cc:2243] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:50.122069 22179 raft_consensus.cc:2272] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:50.124817 22169 raft_consensus.cc:1064] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:50.126885 22179 ts_tablet_manager.cc:1916] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:50.127898 22169 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.130931 22109 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:50.136511 22020 raft_consensus.cc:2955] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.143906 22179 ts_tablet_manager.cc:1929] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:50.144474 22179 log.cc:1199] T 172b9f2bc43b42fca62c756aa5cf17a1 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/172b9f2bc43b42fca62c756aa5cf17a1
I20251214 02:58:50.144933 21031 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 4 to 5, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.146224 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 172b9f2bc43b42fca62c756aa5cf17a1 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:50.153975 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:50.156675 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b9c9a9914df4ca0aa72659e67ec2c4c with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:50.161177 21126 raft_consensus.cc:2955] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.179039 22181 tablet_replica.cc:333] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:50.180480 22181 raft_consensus.cc:2243] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:50.181475 22181 raft_consensus.cc:2272] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:50.185169 22181 ts_tablet_manager.cc:1916] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:50.198683 21277 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:50.198753 22020 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:50.200021 21208 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:50.201305 22090 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:50.202868 22090 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:50.204979 22169 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:50.216656 22181 ts_tablet_manager.cc:1929] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20251214 02:58:50.217145 22181 log.cc:1199] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/2b9c9a9914df4ca0aa72659e67ec2c4c
I20251214 02:58:50.219919 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 2b9c9a9914df4ca0aa72659e67ec2c4c (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:50.233310 22167 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.236882 21274 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.238538 21208 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.245563 22020 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.253162 21029 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.268513 21132 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.273981 21274 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:50.275084 22090 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.276053 22018 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:50.278057 22090 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.282790 22167 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.286396 22018 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.288980 21274 raft_consensus.cc:2955] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.293812 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 75e354ca56944ff1b7775a8309bca568 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:50.302505 21027 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.323823 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 75e354ca56944ff1b7775a8309bca568 with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:50.324743 22181 tablet_replica.cc:333] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:50.325706 22181 raft_consensus.cc:2243] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:50.326399 22181 raft_consensus.cc:2272] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:50.329663 22181 ts_tablet_manager.cc:1916] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:50.336984 22165 raft_consensus.cc:1064] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:50.339597 22165 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.350798 21126 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:50.351532 22181 ts_tablet_manager.cc:1929] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:50.351967 22181 log.cc:1199] T 75e354ca56944ff1b7775a8309bca568 P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/75e354ca56944ff1b7775a8309bca568
I20251214 02:58:50.352231 22018 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:50.352602 22178 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:58:50.353763 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 75e354ca56944ff1b7775a8309bca568 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:50.355204 21274 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:50.355902 22178 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.358388 22119 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:50.368816 22119 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.370601 21274 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.387809 21027 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.387805 21126 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.396821 22018 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.429034 21204 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:50.435482 21274 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:50.435482 22018 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:50.436861 22178 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:50.438194 22165 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:50.444569 22178 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.447049 22018 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.451794 21277 raft_consensus.cc:2955] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.460498 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4b611d0c068c451bb4eeb7eda4823036 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:50.460800 21031 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:50.472685 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b611d0c068c451bb4eeb7eda4823036 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47052
I20251214 02:58:50.474627 22189 tablet_replica.cc:333] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:58:50.475740 22189 raft_consensus.cc:2243] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:50.476414 22189 raft_consensus.cc:2272] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:50.479209 22189 ts_tablet_manager.cc:1916] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:50.494130 22189 ts_tablet_manager.cc:1929] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:58:50.494467 22189 log.cc:1199] T 4b611d0c068c451bb4eeb7eda4823036 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/4b611d0c068c451bb4eeb7eda4823036
I20251214 02:58:50.495755 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet 4b611d0c068c451bb4eeb7eda4823036 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:51.532547 21204 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:51.537719 21274 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:58:51.538295 21171 consensus_peers.cc:597] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cd37022763049b2829947b4a33d6aea. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:51.538324 21132 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:51.539359 22165 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:51.540310 22192 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:51.547067 22165 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.548367 21274 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.548297 21132 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.561863 21029 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.569298 21132 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
W20251214 02:58:51.576079 21096 consensus_peers.cc:597] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1dc028308e3434791fdd7169a0858b0. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:51.578722 21274 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:51.581221 21204 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:51.581413 22109 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:51.582659 22190 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:51.590867 22109 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.596312 21208 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.603828 21274 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.611716 21132 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:51.623111 21027 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.630079 21208 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251214 02:58:51.630836 21096 consensus_peers.cc:597] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ce8bd94962e4bf48e01e9e747adba15. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:51.631155 21274 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:51.632440 22190 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:51.633293 22203 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:51.641626 22203 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.649413 21208 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.651005 21275 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.660159 21204 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:51.660380 21031 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:51.686878 21277 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:58:51.691255 21171 consensus_peers.cc:597] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19fd2760f37a43e2983b015db150e285. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:51.694777 22165 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:51.713406 22165 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.715777 21132 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:58:51.718629 22208 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:51.719139 21277 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.737488 21031 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:51.739279 21132 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:52.092710 22211 ts_tablet_manager.cc:933] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:52.094718 22211 tablet_copy_client.cc:323] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:52.096318 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 5ce8bd94962e4bf48e01e9e747adba15 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:52.096832 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 5ce8bd94962e4bf48e01e9e747adba15 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-5ce8bd94962e4bf48e01e9e747adba15
I20251214 02:58:52.100323 22213 ts_tablet_manager.cc:933] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:52.103871 22213 tablet_copy_client.cc:323] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:52.105472 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 8cd37022763049b2829947b4a33d6aea from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:52.105990 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 8cd37022763049b2829947b4a33d6aea from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-8cd37022763049b2829947b4a33d6aea
I20251214 02:58:52.107138 21143 tablet_copy_source_session.cc:215] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:52.110009 22211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce8bd94962e4bf48e01e9e747adba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:52.117319 21218 tablet_copy_source_session.cc:215] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:52.120457 22213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd37022763049b2829947b4a33d6aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:52.127401 22211 tablet_copy_client.cc:806] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:52.128075 22211 tablet_copy_client.cc:670] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:52.132635 22211 tablet_copy_client.cc:538] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:52.143265 22211 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:52.144665 22213 tablet_copy_client.cc:806] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:52.145282 22213 tablet_copy_client.cc:670] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:52.146624 22215 ts_tablet_manager.cc:933] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:52.152644 22213 tablet_copy_client.cc:538] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:52.162992 22213 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:52.167455 22215 tablet_copy_client.cc:323] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:52.169286 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet e1dc028308e3434791fdd7169a0858b0 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:52.169855 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet e1dc028308e3434791fdd7169a0858b0 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-e1dc028308e3434791fdd7169a0858b0
I20251214 02:58:52.172972 22217 ts_tablet_manager.cc:933] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:52.177563 21143 tablet_copy_source_session.cc:215] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:52.180713 22215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dc028308e3434791fdd7169a0858b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:52.198756 22213 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:52.199700 22213 tablet_bootstrap.cc:492] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:52.200404 22213 ts_tablet_manager.cc:1403] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.008s
I20251214 02:58:52.209692 22217 tablet_copy_client.cc:323] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:52.211915 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 19fd2760f37a43e2983b015db150e285 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:52.212347 22211 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:52.212476 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 19fd2760f37a43e2983b015db150e285 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-19fd2760f37a43e2983b015db150e285
I20251214 02:58:52.213274 22211 tablet_bootstrap.cc:492] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:52.213973 22211 ts_tablet_manager.cc:1403] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.012s
I20251214 02:58:52.217398 22211 raft_consensus.cc:359] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.218200 22211 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:52.203264 22213 raft_consensus.cc:359] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.218840 22211 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.219250 22213 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:52.231160 22213 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.241106 22211 ts_tablet_manager.cc:1434] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.027s user 0.008s sys 0.003s
I20251214 02:58:52.245402 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-5ce8bd94962e4bf48e01e9e747adba15 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:52.245849 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-5ce8bd94962e4bf48e01e9e747adba15 on tablet 5ce8bd94962e4bf48e01e9e747adba15 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:52.220738 21218 tablet_copy_source_session.cc:215] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:52.257404 22213 ts_tablet_manager.cc:1434] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.057s user 0.007s sys 0.007s
I20251214 02:58:52.258528 22217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19fd2760f37a43e2983b015db150e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:52.259243 21217 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-8cd37022763049b2829947b4a33d6aea received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:52.259692 21217 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-8cd37022763049b2829947b4a33d6aea on tablet 8cd37022763049b2829947b4a33d6aea with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:52.222282 22215 tablet_copy_client.cc:806] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:52.265022 22215 tablet_copy_client.cc:670] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:52.275080 22215 tablet_copy_client.cc:538] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:52.280170 22217 tablet_copy_client.cc:806] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:52.280829 22217 tablet_copy_client.cc:670] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:52.283676 22215 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:52.285341 22217 tablet_copy_client.cc:538] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:52.294457 22217 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:52.313077 22215 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:52.313936 22215 tablet_bootstrap.cc:492] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:52.314661 22215 ts_tablet_manager.cc:1403] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.010s
I20251214 02:58:52.317523 22215 raft_consensus.cc:359] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.318325 22215 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:52.319267 22217 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:52.319047 22215 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.320259 22217 tablet_bootstrap.cc:492] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:52.321501 22217 ts_tablet_manager.cc:1403] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.007s
I20251214 02:58:52.321501 22215 ts_tablet_manager.cc:1434] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20251214 02:58:52.324200 22217 raft_consensus.cc:359] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.325412 22217 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:52.326027 22217 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:52.328301 22217 ts_tablet_manager.cc:1434] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20251214 02:58:52.325448 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-e1dc028308e3434791fdd7169a0858b0 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:52.328775 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-e1dc028308e3434791fdd7169a0858b0 on tablet e1dc028308e3434791fdd7169a0858b0 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:52.330014 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-19fd2760f37a43e2983b015db150e285 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:52.330433 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-19fd2760f37a43e2983b015db150e285 on tablet 19fd2760f37a43e2983b015db150e285 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:52.542140 22020 raft_consensus.cc:1217] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:52.568562 22020 raft_consensus.cc:1217] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:52.575266 22020 raft_consensus.cc:1217] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:58:52.634821 22020 raft_consensus.cc:1217] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:58:53.061759 22109 raft_consensus.cc:1064] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:53.064563 22109 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.074015 22020 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.074537 21274 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.075980 22090 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.076665 22109 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.076963 21204 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.081699 22109 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.084048 22192 raft_consensus.cc:1064] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:53.087726 22192 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.097533 22109 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.102183 21204 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.103510 21275 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.107995 22018 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.110841 21274 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.112244 22231 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.115748 22165 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.115242 22018 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.126930 21029 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:53.131942 21132 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.138288 22192 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.141747 22208 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.143180 22020 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.134613 22226 raft_consensus.cc:1064] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:53.150974 22226 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.156663 21132 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:58:53.162021 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 58073.5us
I20251214 02:58:53.173802 21274 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.182964 21204 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.184782 22109 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.187266 21277 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.189509 22190 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.184033 22020 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:53.191576 22190 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:53.217792 22226 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.222070 21204 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.225726 22020 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.228497 21126 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.233404 21031 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.236830 21207 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:53.240171 22190 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.245728 22018 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:53.247622 22240 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.249601 22090 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.257952 22192 raft_consensus.cc:1064] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:53.260803 22018 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.261420 21207 raft_consensus.cc:2955] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.263700 22192 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.257866 21277 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.273751 21208 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.273733 21027 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.285823 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 5ce8bd94962e4bf48e01e9e747adba15 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:53.287796 22020 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:53.288775 21276 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:53.289150 22241 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:53.290012 22231 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:53.300897 21275 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.301719 22165 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.321035 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 19fd2760f37a43e2983b015db150e285 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:53.322317 22020 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.317065 21028 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.323639 22231 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.310199 22018 raft_consensus.cc:2955] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.338446 21031 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.342386 21124 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.343492 21277 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:58:53.344054 22231 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.345894 22208 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.355777 21126 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.377327 21274 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:53.378836 22226 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.385633 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ce8bd94962e4bf48e01e9e747adba15 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50512
I20251214 02:58:53.388724 22018 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:53.389447 22248 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.391055 22240 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:53.394804 21274 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.395138 22241 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.396323 22017 raft_consensus.cc:2955] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.401743 22019 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.409081 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet 19fd2760f37a43e2983b015db150e285 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47052
I20251214 02:58:53.404028 21277 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.431282 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet e1dc028308e3434791fdd7169a0858b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:53.443158 21124 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.447682 22253 tablet_replica.cc:333] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:58:53.450415 21029 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:53.452909 22253 raft_consensus.cc:2243] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:53.466816 22253 raft_consensus.cc:2272] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:53.450017 22252 tablet_replica.cc:333] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:58:53.468845 22252 raft_consensus.cc:2243] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:53.472642 22252 raft_consensus.cc:2272] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:53.473459 22253 ts_tablet_manager.cc:1916] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:53.479799 21031 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:53.484726 22252 ts_tablet_manager.cc:1916] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:53.485551 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet e1dc028308e3434791fdd7169a0858b0 with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:53.489878 22254 tablet_replica.cc:333] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:53.491024 22254 raft_consensus.cc:2243] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:53.491806 22254 raft_consensus.cc:2272] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:53.500939 21208 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:53.506851 22253 ts_tablet_manager.cc:1929] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:58:53.507403 22253 log.cc:1199] T 19fd2760f37a43e2983b015db150e285 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/19fd2760f37a43e2983b015db150e285
I20251214 02:58:53.511160 22252 ts_tablet_manager.cc:1929] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:53.513079 22252 log.cc:1199] T 5ce8bd94962e4bf48e01e9e747adba15 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/5ce8bd94962e4bf48e01e9e747adba15
I20251214 02:58:53.513345 22017 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:53.509660 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet 19fd2760f37a43e2983b015db150e285 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:53.515324 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 5ce8bd94962e4bf48e01e9e747adba15 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:53.515571 22241 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:53.516712 21124 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:58:53.518404 22241 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:58:53.523589 22254 ts_tablet_manager.cc:1916] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:53.524128 22165 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.526717 22017 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.532155 21124 raft_consensus.cc:2955] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.540598 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 8cd37022763049b2829947b4a33d6aea with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:53.542044 21031 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:53.548983 22254 ts_tablet_manager.cc:1929] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:53.549484 22254 log.cc:1199] T e1dc028308e3434791fdd7169a0858b0 P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/e1dc028308e3434791fdd7169a0858b0
I20251214 02:58:53.551600 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet e1dc028308e3434791fdd7169a0858b0 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:53.556399 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 8cd37022763049b2829947b4a33d6aea with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50512
I20251214 02:58:53.557282 22252 tablet_replica.cc:333] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:58:53.558248 22252 raft_consensus.cc:2243] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:53.558933 22252 raft_consensus.cc:2272] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:53.561622 22252 ts_tablet_manager.cc:1916] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:53.572403 22252 ts_tablet_manager.cc:1929] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:58:53.572777 22252 log.cc:1199] T 8cd37022763049b2829947b4a33d6aea P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/8cd37022763049b2829947b4a33d6aea
I20251214 02:58:53.574453 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 8cd37022763049b2829947b4a33d6aea (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:54.598600 21282 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:54.604130 21206 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251214 02:58:54.604307 21245 consensus_peers.cc:597] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae2c1119874240b6af01af115f82ecd3. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:54.604349 21132 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.606043 22268 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.606822 22232 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.615013 22268 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.616568 21200 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.617422 21132 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.635196 21027 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:54.642474 21132 consensus_queue.cc:237] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
W20251214 02:58:54.650879 21096 consensus_peers.cc:597] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ba4da4df7bc466aa9cc42fa304335d8. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:54.651615 21282 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.652873 22265 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.653513 21200 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.654853 22262 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.686061 22265 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.688033 21200 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.692065 21282 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.711540 21031 catalog_manager.cc:5654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.710139 21132 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:54.726475 21273 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251214 02:58:54.727228 21096 consensus_peers.cc:597] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a4a2d7bee4345bbb9b7631ce9e5d5de. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:54.727490 21206 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.730218 22248 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.731149 22259 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.742127 22259 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.743623 21206 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.748706 21273 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.760372 21132 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:54.761102 21031 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
W20251214 02:58:54.785991 21096 consensus_peers.cc:597] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ddd57244a03440195bc9142e5d4344a. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:54.786861 21206 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.788657 21282 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:54.789112 22263 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.797358 22248 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:54.810286 22259 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.812165 21200 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.813845 21282 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:54.830833 21031 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:55.064795 22285 ts_tablet_manager.cc:933] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:55.066941 22285 tablet_copy_client.cc:323] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:55.068188 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 4ba4da4df7bc466aa9cc42fa304335d8 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:55.068694 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 4ba4da4df7bc466aa9cc42fa304335d8 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-4ba4da4df7bc466aa9cc42fa304335d8
I20251214 02:58:55.075019 21143 tablet_copy_source_session.cc:215] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:55.078696 22285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba4da4df7bc466aa9cc42fa304335d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:55.091701 22285 tablet_copy_client.cc:806] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:55.092289 22285 tablet_copy_client.cc:670] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:55.096926 22285 tablet_copy_client.cc:538] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:55.104331 22285 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:55.127358 22288 ts_tablet_manager.cc:933] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:55.129199 22288 tablet_copy_client.cc:323] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:58:55.130825 21292 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet ae2c1119874240b6af01af115f82ecd3 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:58:55.131349 21292 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet ae2c1119874240b6af01af115f82ecd3 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-ae2c1119874240b6af01af115f82ecd3
I20251214 02:58:55.131609 22285 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:55.132499 22285 tablet_bootstrap.cc:492] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:55.133193 22285 ts_tablet_manager.cc:1403] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.010s
I20251214 02:58:55.136014 22285 raft_consensus.cc:359] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.136842 22285 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:55.137496 22285 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.140619 22285 ts_tablet_manager.cc:1434] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20251214 02:58:55.141894 21292 tablet_copy_source_session.cc:215] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:55.142838 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-4ba4da4df7bc466aa9cc42fa304335d8 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:55.143288 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-4ba4da4df7bc466aa9cc42fa304335d8 on tablet 4ba4da4df7bc466aa9cc42fa304335d8 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:55.154829 22288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c1119874240b6af01af115f82ecd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:55.171736 22288 tablet_copy_client.cc:806] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:55.172228 22288 tablet_copy_client.cc:670] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:55.175195 22288 tablet_copy_client.cc:538] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:55.183699 22288 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
W20251214 02:58:55.185860 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800150980 after lost signal to thread 19984
W20251214 02:58:55.186905 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800150c40 after lost signal to thread 21065
W20251214 02:58:55.188019 21154 debug-util.cc:398] Leaking SignalData structure 0x7b080028d620 after lost signal to thread 21157
W20251214 02:58:55.189221 21154 debug-util.cc:398] Leaking SignalData structure 0x7b080028da80 after lost signal to thread 21232
W20251214 02:58:55.190155 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8c60 after lost signal to thread 21306
W20251214 02:58:55.190986 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800289ba0 after lost signal to thread 22044
I20251214 02:58:55.196941 22285 ts_tablet_manager.cc:933] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:55.205888 22285 tablet_copy_client.cc:323] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:55.207974 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:55.208509 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-9a4a2d7bee4345bbb9b7631ce9e5d5de
I20251214 02:58:55.212421 22288 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:55.213323 22288 tablet_bootstrap.cc:492] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:55.213986 22288 ts_tablet_manager.cc:1403] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.016s
I20251214 02:58:55.217572 22288 raft_consensus.cc:359] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.218640 22288 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:55.219290 22288 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.221804 21143 tablet_copy_source_session.cc:215] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:55.224239 22288 ts_tablet_manager.cc:1434] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20251214 02:58:55.224824 22285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:55.227299 21292 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-ae2c1119874240b6af01af115f82ecd3 received from {username='slave'} at 127.0.0.1:50140
I20251214 02:58:55.227684 21292 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-ae2c1119874240b6af01af115f82ecd3 on tablet ae2c1119874240b6af01af115f82ecd3 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:55.244056 22285 tablet_copy_client.cc:806] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:55.244688 22285 tablet_copy_client.cc:670] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:55.249117 22285 tablet_copy_client.cc:538] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:55.261945 22285 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:55.285734 22285 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:55.286888 22285 tablet_bootstrap.cc:492] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:55.287776 22285 ts_tablet_manager.cc:1403] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.004s
I20251214 02:58:55.290498 22285 raft_consensus.cc:359] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.291322 22285 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:55.292038 22285 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.301781 22285 ts_tablet_manager.cc:1434] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251214 02:58:55.309427 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-9a4a2d7bee4345bbb9b7631ce9e5d5de received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:55.309854 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-9a4a2d7bee4345bbb9b7631ce9e5d5de on tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:55.374117 22285 ts_tablet_manager.cc:933] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:55.390678 22285 tablet_copy_client.cc:323] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:55.392612 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 0ddd57244a03440195bc9142e5d4344a from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:55.393189 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 0ddd57244a03440195bc9142e5d4344a from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-0ddd57244a03440195bc9142e5d4344a
I20251214 02:58:55.400681 21143 tablet_copy_source_session.cc:215] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:55.403879 22285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ddd57244a03440195bc9142e5d4344a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:55.425542 22285 tablet_copy_client.cc:806] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:55.426160 22285 tablet_copy_client.cc:670] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:55.429418 22285 tablet_copy_client.cc:538] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:55.436178 22285 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:55.455520 22285 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:55.456532 22285 tablet_bootstrap.cc:492] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:55.457144 22285 ts_tablet_manager.cc:1403] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20251214 02:58:55.460337 22285 raft_consensus.cc:359] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.462324 22285 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:55.462975 22285 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:55.465962 22285 ts_tablet_manager.cc:1434] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20251214 02:58:55.467480 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-0ddd57244a03440195bc9142e5d4344a received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:55.467873 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-0ddd57244a03440195bc9142e5d4344a on tablet 0ddd57244a03440195bc9142e5d4344a with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:55.533725 22018 raft_consensus.cc:1217] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:55.612423 22018 raft_consensus.cc:1217] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:55.747061 22018 raft_consensus.cc:1217] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:55.992338 22018 raft_consensus.cc:1217] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:56.074244 22255 raft_consensus.cc:1064] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:56.077122 22255 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.085110 22018 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.085314 21200 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.086783 22298 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.088727 22298 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.095741 21126 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.097594 22255 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.107514 22298 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.112583 21200 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.122975 22018 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.135311 21126 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.136543 21029 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.146710 22265 raft_consensus.cc:1064] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:56.149837 22265 consensus_queue.cc:237] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.177124 21273 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.178727 22240 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.170979 22018 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.186292 21200 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.188335 22240 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.189273 22265 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.215708 22248 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.223222 21200 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.218353 22018 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.244320 21273 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.280807 22311 raft_consensus.cc:1064] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:56.282792 21282 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.283243 22311 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.284452 21031 catalog_manager.cc:5654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.302852 22017 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.305213 22248 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.307473 21276 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.308872 22248 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.309381 21200 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.311179 22248 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.314378 22017 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.316483 21200 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.317343 22298 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.319900 22255 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.332319 22297 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.333287 21200 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.336786 22018 raft_consensus.cc:2955] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.339725 22311 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.344782 22017 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.353402 21200 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.367307 21276 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.370087 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet ae2c1119874240b6af01af115f82ecd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:56.376914 21029 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:58:56.381737 21017 catalog_manager.cc:5177] ChangeConfig:REMOVE_PEER RPC for tablet ae2c1119874240b6af01af115f82ecd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20251214 02:58:56.379464 21124 consensus_queue.cc:237] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.392244 21028 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:56.408524 22017 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.411414 22248 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.418045 21273 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.421636 22310 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.425825 21124 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.429366 22248 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.436764 22018 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.449136 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4ba4da4df7bc466aa9cc42fa304335d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:56.449734 21273 raft_consensus.cc:2955] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.456990 21031 catalog_manager.cc:5654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:56.459209 21206 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.462956 22259 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.469426 22240 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.473945 21206 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.479944 22018 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.484501 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ba4da4df7bc466aa9cc42fa304335d8 with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:56.485607 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet ae2c1119874240b6af01af115f82ecd3 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47052
I20251214 02:58:56.502049 22259 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.505069 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:56.508235 22317 tablet_replica.cc:333] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:56.516132 22317 raft_consensus.cc:2243] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:56.518082 22317 raft_consensus.cc:2272] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:56.524679 22317 ts_tablet_manager.cc:1916] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:56.525435 22318 tablet_replica.cc:333] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:58:56.517397 22018 raft_consensus.cc:2955] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.542358 22318 raft_consensus.cc:2243] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:56.542939 22317 ts_tablet_manager.cc:1929] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:56.543514 22317 log.cc:1199] T 4ba4da4df7bc466aa9cc42fa304335d8 P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/4ba4da4df7bc466aa9cc42fa304335d8
I20251214 02:58:56.543077 22318 raft_consensus.cc:2272] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:56.548970 22318 ts_tablet_manager.cc:1916] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:56.549896 22265 raft_consensus.cc:1064] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:56.551956 22265 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.546106 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 4ba4da4df7bc466aa9cc42fa304335d8 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:56.568341 22318 ts_tablet_manager.cc:1929] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:56.568899 22318 log.cc:1199] T ae2c1119874240b6af01af115f82ecd3 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/ae2c1119874240b6af01af115f82ecd3
I20251214 02:58:56.571419 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet ae2c1119874240b6af01af115f82ecd3 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:56.574821 22017 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.576457 21200 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.577247 21275 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:56.579080 22265 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.580602 22259 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.597743 22265 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:56.593878 21031 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.625849 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50512
I20251214 02:58:56.634749 22259 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.636098 21206 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.644317 21273 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.643090 22018 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.653980 21028 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.664069 22323 tablet_replica.cc:333] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:58:56.665020 22323 raft_consensus.cc:2243] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:56.670208 22323 raft_consensus.cc:2272] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:56.675784 22323 ts_tablet_manager.cc:1916] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:56.689287 21126 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:56.699908 21275 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.703298 22311 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.711346 22017 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:58:56.711731 22248 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.715301 22265 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:56.715930 21275 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.719071 22323 ts_tablet_manager.cc:1929] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:56.719548 22323 log.cc:1199] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/9a4a2d7bee4345bbb9b7631ce9e5d5de
I20251214 02:58:56.722304 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 9a4a2d7bee4345bbb9b7631ce9e5d5de (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:56.727123 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 0ddd57244a03440195bc9142e5d4344a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:58:56.730594 22017 raft_consensus.cc:2955] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:56.735107 21031 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 1 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:56.748847 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ddd57244a03440195bc9142e5d4344a with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20251214 02:58:56.749759 22317 tablet_replica.cc:333] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:58:56.750754 22317 raft_consensus.cc:2243] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:58:56.751495 22317 raft_consensus.cc:2272] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:58:56.754873 22317 ts_tablet_manager.cc:1916] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:58:56.770476 22317 ts_tablet_manager.cc:1929] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:58:56.770896 22317 log.cc:1199] T 0ddd57244a03440195bc9142e5d4344a P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/0ddd57244a03440195bc9142e5d4344a
I20251214 02:58:56.772173 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 0ddd57244a03440195bc9142e5d4344a (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:58:57.797130 21200 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:57.803767 21273 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:58:57.804095 21126 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20251214 02:58:57.807964 21171 consensus_peers.cc:597] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75c047f1d6d94237afc050db2241d659. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:57.809409 22294 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:57.810192 22264 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:57.819418 22294 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.825778 21126 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.836318 21273 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.845260 21031 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.859905 21206 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:57.888686 21275 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:57.890197 22264 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:58:57.891005 21171 consensus_peers.cc:597] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e18f39bed60f44f5b8c25ede5d8bba8f. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:57.895249 21126 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:57.896551 22264 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:58:57.900960 22294 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.919152 21273 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.947849 21132 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:57.960736 21126 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.971482 21278 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:58:57.971829 21096 consensus_peers.cc:597] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b5a42ee941246bf8d0050ea0dcfb9fc. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:57.973146 21200 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:58:57.975220 22259 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:57.975883 22248 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:57.985558 22259 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.991235 21206 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:57.994376 21029 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:58.017799 21273 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.029984 21031 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:58.029973 21275 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:58.048115 21206 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 02:58:58.051492 21126 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251214 02:58:58.053421 21245 consensus_peers.cc:597] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf5cb5c0f4854e4a846c25eb5ce80a9a. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:58:58.054049 22298 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:58.075441 22298 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:58:58.077361 22344 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:58.081535 21200 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:58.093302 21126 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:58:58.107537 21031 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 2, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:58:58.316747 22348 ts_tablet_manager.cc:933] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:58.320928 22348 tablet_copy_client.cc:323] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:58.326597 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 75c047f1d6d94237afc050db2241d659 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:58.327405 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 75c047f1d6d94237afc050db2241d659 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-75c047f1d6d94237afc050db2241d659
I20251214 02:58:58.337610 21218 tablet_copy_source_session.cc:215] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:58.348044 22348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c047f1d6d94237afc050db2241d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:58.378582 22348 tablet_copy_client.cc:806] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:58.379348 22348 tablet_copy_client.cc:670] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:58.386251 22348 tablet_copy_client.cc:538] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:58.395509 22348 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:58.420147 22348 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:58.420913 22348 tablet_bootstrap.cc:492] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:58.421449 22348 ts_tablet_manager.cc:1403] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.002s
I20251214 02:58:58.424345 22348 raft_consensus.cc:359] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.425734 22348 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:58.426640 22348 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.431020 22348 ts_tablet_manager.cc:1434] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20251214 02:58:58.434830 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-75c047f1d6d94237afc050db2241d659 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:58.435276 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-75c047f1d6d94237afc050db2241d659 on tablet 75c047f1d6d94237afc050db2241d659 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:58.448139 22348 ts_tablet_manager.cc:933] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:58:58.450711 22348 tablet_copy_client.cc:323] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:58:58.452056 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet e18f39bed60f44f5b8c25ede5d8bba8f from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:58:58.452538 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet e18f39bed60f44f5b8c25ede5d8bba8f from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-e18f39bed60f44f5b8c25ede5d8bba8f
I20251214 02:58:58.460623 21218 tablet_copy_source_session.cc:215] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:58.463966 22348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18f39bed60f44f5b8c25ede5d8bba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:58.483223 22348 tablet_copy_client.cc:806] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:58.483852 22348 tablet_copy_client.cc:670] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:58.486259 22353 ts_tablet_manager.cc:933] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:58:58.489130 22348 tablet_copy_client.cc:538] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:58.502558 22348 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:58.517737 22353 tablet_copy_client.cc:323] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:58:58.519910 21292 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:58:58.520498 21292 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet bf5cb5c0f4854e4a846c25eb5ce80a9a from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-bf5cb5c0f4854e4a846c25eb5ce80a9a
I20251214 02:58:58.529114 21292 tablet_copy_source_session.cc:215] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:58.532423 22353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:58.549731 22353 tablet_copy_client.cc:806] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:58.550357 22353 tablet_copy_client.cc:670] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:58.561545 22353 tablet_copy_client.cc:538] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:58.574607 22353 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:58.624665 22353 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:58.625667 22353 tablet_bootstrap.cc:492] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:58.626397 22353 ts_tablet_manager.cc:1403] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.052s user 0.026s sys 0.004s
I20251214 02:58:58.629539 22353 raft_consensus.cc:359] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.630414 22353 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:58.631284 22353 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.635876 22353 ts_tablet_manager.cc:1434] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251214 02:58:58.645605 21292 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-bf5cb5c0f4854e4a846c25eb5ce80a9a received from {username='slave'} at 127.0.0.1:50140
I20251214 02:58:58.646063 21292 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-bf5cb5c0f4854e4a846c25eb5ce80a9a on tablet bf5cb5c0f4854e4a846c25eb5ce80a9a with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:58.655756 22356 ts_tablet_manager.cc:933] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:58:58.666174 22348 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:58.667116 22348 tablet_bootstrap.cc:492] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:58.667869 22348 ts_tablet_manager.cc:1403] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.166s user 0.070s sys 0.053s
I20251214 02:58:58.671434 22348 raft_consensus.cc:359] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.673223 22348 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:58.673902 22348 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.676784 22348 ts_tablet_manager.cc:1434] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251214 02:58:58.687906 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-e18f39bed60f44f5b8c25ede5d8bba8f received from {username='slave'} at 127.0.0.1:47944
I20251214 02:58:58.688318 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-e18f39bed60f44f5b8c25ede5d8bba8f on tablet e18f39bed60f44f5b8c25ede5d8bba8f with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:58.713474 22356 tablet_copy_client.cc:323] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:58:58.715369 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:58:58.715930 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 6b5a42ee941246bf8d0050ea0dcfb9fc from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-6b5a42ee941246bf8d0050ea0dcfb9fc
I20251214 02:58:58.724373 21143 tablet_copy_source_session.cc:215] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:58:58.729182 22356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:58:58.744014 22356 tablet_copy_client.cc:806] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:58:58.744557 22356 tablet_copy_client.cc:670] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:58:58.750641 22356 tablet_copy_client.cc:538] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:58:58.759418 22356 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:58:58.793704 22356 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:58:58.796056 22356 tablet_bootstrap.cc:492] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:58:58.796805 22356 ts_tablet_manager.cc:1403] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.005s
I20251214 02:58:58.799610 22356 raft_consensus.cc:359] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.800226 22356 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:58:58.800920 22356 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:58:58.803308 22356 ts_tablet_manager.cc:1434] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251214 02:58:58.805223 21143 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-6b5a42ee941246bf8d0050ea0dcfb9fc received from {username='slave'} at 127.0.0.1:52098
I20251214 02:58:58.805568 21143 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-6b5a42ee941246bf8d0050ea0dcfb9fc on tablet 6b5a42ee941246bf8d0050ea0dcfb9fc with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:58:58.930518 22018 raft_consensus.cc:1217] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251214 02:58:59.009884 22018 raft_consensus.cc:1217] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:59.036718 22018 raft_consensus.cc:1217] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251214 02:58:59.116959 22018 raft_consensus.cc:1217] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251214 02:58:59.357354 22362 raft_consensus.cc:1064] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:59.363607 22362 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:59.462997 22344 raft_consensus.cc:1064] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:59.466136 22344 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:59.509269 22294 raft_consensus.cc:1064] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:58:59.511636 22294 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:58:59.566423 22018 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:59.567915 22345 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:59.597083 21200 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:59.634734 22345 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:58:59.636976 22018 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:58:59.640946 22362 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:58:59.690299 21282 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:58:59.733724 22020 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:59.737943 22325 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251214 02:58:59.777773 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 109658us
I20251214 02:58:59.785226 22361 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:58:59.801468 21278 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:58:59.819180 22370 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:58:59.836110 22367 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:59.837592 21278 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:59.851482 22018 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:59.856571 21031 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:58:59.859828 22344 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:58:59.861588 21199 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:00.356715 22371 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.357420 22371 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.359434 22371 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:00.403537 21278 consensus_meta.cc:287] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.003s
I20251214 02:59:00.413144 22371 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.413776 22371 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.415825 22381 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.416356 22381 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.416826 22371 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.418294 22381 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.419450 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.426487 22344 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.427065 22344 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.428279 22375 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.428787 22375 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.429122 22344 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.430243 22375 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.431041 22375 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.431806 22344 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.432400 22344 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:00.434398 22344 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.431613 22375 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:00.440186 22375 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.441844 22375 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.442210 22375 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.444101 22375 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.446908 22325 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.447431 22325 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.448948 22325 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.450424 22294 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.450850 22294 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.452385 22294 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.454058 22294 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.454366 22390 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.454463 22294 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.454918 22390 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.456166 22294 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.456681 22390 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:00.457970 21199 consensus_meta.cc:287] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.595s user 0.004s sys 0.008s
I20251214 02:59:00.471580 22344 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.471797 22345 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.472129 22344 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.472276 22345 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.474063 22345 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.474465 22344 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:00.472280 22388 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.476632 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.476842 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.477118 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.476084 22388 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.478327 22361 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.478825 22361 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.480659 22361 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.483317 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.484545 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.485055 21274 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:00.485091 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.487416 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.491710 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.495249 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.495860 21280 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:00.496964 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.498468 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.499073 21281 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:00.501096 22345 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.503132 22388 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.504802 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:00.507143 22375 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.512903 22377 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.513355 22375 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.513422 22377 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.515257 22389 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:00.515537 22375 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.515666 22389 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.516050 22389 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:00.516188 22377 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.516969 22377 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.517422 22377 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.518429 22375 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.518965 22375 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.519253 22377 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.521813 21124 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:59:00.521929 22381 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.509236 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.504494 22388 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.512444 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.524868 21200 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:00.522469 22381 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.526273 22388 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.526937 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.529656 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.530104 21200 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:00.538137 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.528118 22375 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.540040 22375 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.527484 22381 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.545607 22381 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.546103 22381 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.548028 22381 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.558730 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.564383 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:00.564992 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.567819 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.568468 21201 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:00.569725 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:00.570940 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.571278 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.550513 22017 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:00.570676 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.555862 22389 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.560956 22373 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:00.572770 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.554919 22375 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.565155 22381 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.574237 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.575007 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.575223 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.575914 22396 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.577723 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.577373 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.583042 22373 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.589046 21248 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:00.591850 21200 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:00.589843 22325 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:00.597733 22325 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.598105 22325 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:00.604094 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.604557 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.577311 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.589717 22396 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.615024 22373 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.615614 22345 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.616618 22396 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.617503 22345 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.617645 22396 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:00.617992 22396 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.618360 22396 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:00.627733 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:00.627640 22380 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.618005 22345 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.628862 22388 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.590262 22381 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.632491 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:00.628423 22340 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:00.638808 22345 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.633049 22380 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.641599 22345 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.641947 22380 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.642127 22345 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:00.643074 22345 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.639827 22394 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.642735 22340 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:00.648048 22397 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.648993 22375 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:00.650143 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:00.659448 22397 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.643455 22345 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:00.651087 22381 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.643957 22388 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20251214 02:59:00.664323 22375 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.664913 22375 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.666684 22381 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.666872 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.667199 22381 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.669063 22381 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.669567 22340 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.653707 22325 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.656167 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.658800 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.658928 22389 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.659333 22394 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.658226 22396 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.663079 22397 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.663611 22395 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.674800 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:00.679322 22395 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.677824 22325 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.678529 22394 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.692029 22375 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.699047 22294 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.692497 22396 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.678740 22345 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.703776 22381 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:00.706792 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 137145us
I20251214 02:59:00.704869 22381 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.713639 22381 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.714129 22381 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.710242 22294 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.716274 22294 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:00.716814 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 116076us
I20251214 02:59:00.718320 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:00.719131 22394 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:00.719570 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:00.720432 22392 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:00.720732 21247 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:00.721411 22392 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:00.724736 22294 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.725200 22294 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.726928 22294 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.727694 22294 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.718196 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:00.721822 21247 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:00.729563 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:00.730211 21199 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:00.728693 22018 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:00.731582 22392 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:00.728228 22294 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.731950 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:00.732849 22392 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:00.733186 22392 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.733352 22294 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.703438 22345 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.717041 22395 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.733541 22392 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:00.736373 22344 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:00.747956 21199 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:00.749325 22401 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.749564 22399 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.748384 22340 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.750653 22394 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.750110 22399 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.750279 22340 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.749878 22345 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:00.750118 22401 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.752724 22345 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.754122 22401 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.755646 22382 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.755548 22373 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:00.729821 22381 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:00.756196 22382 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:00.756655 22373 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:00.757894 22373 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:00.757979 22345 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.759294 22382 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.760577 22402 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.760989 22402 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.758893 22373 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.763484 22402 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.764932 22376 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:00.765247 22381 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.766172 22402 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:00.757268 22394 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.766595 22402 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:00.768616 22394 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:00.769080 22373 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:00.770112 22399 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:00.770476 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 92933.2us
I20251214 02:59:00.773329 22340 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.775792 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:00.777524 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:00.770013 22402 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.906481 22384 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.778913 22382 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:00.914628 22405 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:00.914968 22377 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:00.916227 22377 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.918828 22377 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:00.925988 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 167259us
I20251214 02:59:00.931456 21028 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:01.054172 22294 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:01.054852 22294 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:00.907428 22392 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:01.094100 22392 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:00.943109 22402 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:01.416114 22294 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:00.909343 22384 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:01.545611 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800321400 after lost signal to thread 19984
W20251214 02:59:01.546829 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800327ee0 after lost signal to thread 21065
W20251214 02:59:01.547974 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0200 after lost signal to thread 21157
W20251214 02:59:01.548951 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08001a9fe0 after lost signal to thread 21232
W20251214 02:59:01.549968 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800305d20 after lost signal to thread 21306
I20251214 02:59:01.628455 22384 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:01.656522 22373 consensus_meta.cc:287] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.886s user 0.007s sys 0.000s
W20251214 02:59:01.442149 21097 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.026s sys 0.064s
W20251214 02:59:01.754050 21097 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s user 0.027s sys 0.065s
I20251214 02:59:01.779917 22373 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.780695 22373 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:01.810370 22402 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.811198 22402 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.813512 22402 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.824246 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.826208 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.827867 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:01.874861 21096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.004s sys 0.015s
W20251214 02:59:01.875428 21096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.004s sys 0.015s
I20251214 02:59:01.883709 22407 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.884402 22407 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.886505 22407 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:01.887252 21028 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.919s user 0.002s sys 0.012s
W20251214 02:59:01.887650 21028 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.943s user 0.003s sys 0.012s
W20251214 02:59:01.888301 21098 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.013s sys 0.029s
W20251214 02:59:01.888684 21098 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.979s user 0.013s sys 0.029s
W20251214 02:59:01.889250 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 1.1118e+06us
I20251214 02:59:01.890270 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:01.892073 22377 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:01.913920 21200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1658) took 1377 ms. Trace:
I20251214 02:59:01.914465 21200 rpcz_store.cc:276] 1214 02:59:00.536713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:00.536832 (+ 119us) service_pool.cc:225] Handling call
1214 02:59:00.540957 (+ 4125us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:00.592226 (+ 51269us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:01.913601 (+1321375us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x7b5400147dd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:01.913820 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114702592}
W20251214 02:59:01.927850 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.003s sys 0.008s
W20251214 02:59:01.931393 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.003s sys 0.008s
W20251214 02:59:01.928756 21171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.003s sys 0.009s
I20251214 02:59:01.929311 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:01.936200 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.003s sys 0.009s
I20251214 02:59:01.930385 22411 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.937528 22411 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.938989 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.939143 22411 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.940404 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.940738 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.941267 22373 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.941938 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:01.943385 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08002cca20 after lost signal to thread 22044
I20251214 02:59:01.945955 21124 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:01.957528 22340 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.958395 21130 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251214 02:59:01.958995 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.959554 21124 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:01.959712 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.959879 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.960132 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.958067 22340 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.961236 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.959475 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.961472 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.961941 21128 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:01.962212 22340 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.962817 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.963411 22340 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:01.964121 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:01.963915 22340 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:01.964898 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.965122 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:01.966022 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:01.966840 22340 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.966063 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.970870 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.971577 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.972702 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.973250 21130 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
W20251214 02:59:01.975379 21248 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.384s user 0.010s sys 0.015s
W20251214 02:59:01.975946 21248 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.385s user 0.011s sys 0.015s
W20251214 02:59:01.976488 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 1.44708e+06us
I20251214 02:59:01.977162 21248 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:01.977331 21133 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:01.980036 22410 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:01.981940 22410 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.983963 22410 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:01.981326 22389 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:01.984568 22389 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:01.985266 22398 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:01.985754 22398 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:01.986411 22344 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:01.986383 22389 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:01.987865 22344 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:01.988227 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.989712 22344 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:01.991099 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.991046 22398 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:01.992197 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.992122 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.993702 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.994081 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1660) took 1455 ms. Trace:
I20251214 02:59:01.994115 22398 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:01.994551 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:01.995307 22398 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:01.995676 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:01.995811 22398 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:01.994451 21207 rpcz_store.cc:276] 1214 02:59:00.538257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:00.538498 (+ 241us) service_pool.cc:225] Handling call
1214 02:59:01.993458 (+1454960us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:01.993643 (+ 185us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:01.993803 (+ 160us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:01.993828 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:01.993865 (+ 37us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:01.993892 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:01.993942 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:01.994033 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4680064}
I20251214 02:59:01.996068 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.996282 22398 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:01.997460 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:01.997771 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:01.998287 22344 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.016533 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.001363 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:02.018206 21202 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:02.016345 22415 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.002179 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:01.995819 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:02.020859 21128 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:02.001398 22392 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.003858 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 1641) took 1402 ms. Trace:
I20251214 02:59:01.998507 21207 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:02.024580 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.025110 21202 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:02.025475 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.025130 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.027110 21199 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:02.019238 22415 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.020615 22398 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.029096 22415 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.029165 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.002359 22394 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.029888 21199 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:02.030009 22394 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.030319 22398 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.023113 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.005060 22345 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.031177 21200 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:02.022945 21278 rpcz_store.cc:276] 1214 02:59:00.601556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:00.601673 (+ 117us) service_pool.cc:225] Handling call
1214 02:59:00.601964 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:00.677045 (+ 75081us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:00.677073 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:00.677119 (+ 46us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:00.677147 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:02.003599 (+1326452us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b5400388bd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:02.003649 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:02.003774 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172271872}
I20251214 02:59:02.022070 22392 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.032501 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.033416 22394 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.033525 22392 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.034261 22345 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.036176 22345 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.039316 22394 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.041070 22394 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.042690 22418 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.042970 22415 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.043179 22392 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.043660 22418 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.045818 22392 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.046154 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.046916 21278 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.048570 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 206) took 1316 ms. Trace:
I20251214 02:59:02.049811 21031 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 202) took 1347 ms. Trace:
I20251214 02:59:02.050535 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.051337 22376 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.052218 22376 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.054502 22376 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.056413 22398 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.056911 22398 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.057320 22392 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.050199 21031 rpcz_store.cc:276] 1214 02:59:00.702004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:00.704255 (+ 2251us) service_pool.cc:225] Handling call
1214 02:59:02.049743 (+1345488us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164802,"reported_tablets":2}
I20251214 02:59:02.058849 22398 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.061189 22418 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.043573 22382 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.061955 22382 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.063989 22382 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.043941 22415 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.065891 22415 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.066915 22345 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.070060 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.071033 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.071596 21273 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.070649 21279 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:59:02.072290 22389 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.074951 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.075098 22389 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:02.075505 21276 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:59:02.077117 22389 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.078351 22388 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.070088 22345 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.078940 22388 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.079385 22389 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.079864 22389 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.078452 22416 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.083518 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.083649 22416 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:02.084327 22398 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.086885 21276 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.077637 22418 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.080300 22345 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.077803 22415 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.090420 22410 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.090772 22416 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.090989 22410 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.091392 22388 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.092759 22410 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.093008 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.082803 22392 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.094617 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.094370 22392 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.095769 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.096084 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:02.098064 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.099186 22389 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.096735 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.099920 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.100274 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.096499 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.098117 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.101224 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.048913 21028 rpcz_store.cc:276] 1214 02:59:00.732465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:00.732612 (+ 147us) service_pool.cc:225] Handling call
1214 02:59:02.048485 (+1315873us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:01.748441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:01.749026 (+ 585us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:01.749062 (+ 36us) write_op.cc:435] Acquired schema lock
1214 02:59:01.749079 (+ 17us) tablet.cc:660] Decoding operations
1214 02:59:01.749434 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:01.749509 (+ 75us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:01.749538 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:01.749723 (+ 185us) tablet.cc:699] Row locks acquired
1214 02:59:01.749744 (+ 21us) write_op.cc:260] PREPARE: finished
1214 02:59:01.749897 (+ 153us) write_op.cc:270] Start()
1214 02:59:01.750007 (+ 110us) write_op.cc:276] Timestamp: P: 1765681141749848 usec, L: 0
1214 02:59:01.750027 (+ 20us) op_driver.cc:348] REPLICATION: starting
1214 02:59:01.928314 (+178287us) log.cc:844] Serialized 598 byte log entry
1214 02:59:01.942307 (+ 13993us) op_driver.cc:464] REPLICATION: finished
1214 02:59:01.952870 (+ 10563us) write_op.cc:301] APPLY: starting
1214 02:59:01.952946 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:01.953038 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:01.953156 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:01.953386 (+ 230us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:01.953530 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:01.953571 (+ 41us) write_op.cc:312] APPLY: finished
1214 02:59:01.953739 (+ 168us) log.cc:844] Serialized 22 byte log entry
1214 02:59:01.954772 (+ 1033us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:01.954850 (+ 78us) write_op.cc:454] Released schema lock
1214 02:59:01.954885 (+ 35us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:01.956647 (+ 1762us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:01.956682 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":10333,"apply.run_cpu_time_us":1467,"apply.run_wall_time_us":3773,"num_ops":1,"prepare.queue_time_us":804178,"prepare.run_cpu_time_us":3074,"prepare.run_wall_time_us":192587,"raft.queue_time_us":6905,"replication_time_us":192163,"spinlock_wait_cycles":7626112,"thread_start_us":974943,"threads_started":4,"wal-append.queue_time_us":1984}]]}
I20251214 02:59:02.103688 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.104892 22388 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.105412 22388 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.087785 22418 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.109009 22388 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.109827 22418 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.113361 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.114092 21276 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.111726 22392 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.096064 22415 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.114742 22394 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.115391 22388 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.139110 22388 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.140951 22388 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.118448 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.141765 22016 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:02.125813 21281 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.133967 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.146608 22415 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.148178 22344 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.148373 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.148687 22344 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.149000 21276 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:02.151348 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.151880 21276 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.153046 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.153800 21276 raft_consensus.cc:2410] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20251214 02:59:02.155889 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.156435 21277 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.158835 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.161010 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.161865 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.160041 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:02.167210 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.171317 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.172166 21279 raft_consensus.cc:2410] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20251214 02:59:02.177623 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:02.199424 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.201828 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.203469 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.204181 21275 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:02.161350 21127 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:02.170938 22396 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.208843 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:02.209362 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.177676 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:02.207774 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.211571 21098 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:02.214979 22396 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.216390 22396 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.217422 22361 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.217922 22361 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.218443 22423 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.218708 22389 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.219194 22371 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.219043 22423 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.219673 22371 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.221608 22371 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.223615 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:02.224332 21273 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:02.224694 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.225001 22389 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.226095 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.226890 22389 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.227952 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.228502 22018 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:02.229104 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.229691 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.230235 22018 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:02.226020 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.230926 21274 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:02.231200 21171 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:02.231918 22389 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:02.232093 22361 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.221710 22423 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.232362 22389 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.233883 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.245791 22389 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:02.235213 22396 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.215397 21098 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:02.257965 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:02.215667 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.236932 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.158437 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.237612 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.239822 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.241624 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.243058 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.245784 22374 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.254113 22389 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.254861 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.256210 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.259271 21276 raft_consensus.cc:2393] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:02.260591 21199 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.263055 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.214759 22420 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.270872 22402 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.272394 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.273343 22389 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.273628 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.274675 22419 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.272567 22377 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.256951 22340 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251214 02:59:02.275358 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:02.276353 22018 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:02.270397 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.277904 21207 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.275359 22377 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.280076 22377 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.280227 22388 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.280301 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.295708 22018 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:02.289949 21248 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.296766 21171 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:02.296996 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.278772 22374 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.297951 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.301446 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.302816 21199 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:02.305549 21248 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.295130 22388 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.307099 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.309484 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.307822 22394 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.312224 22394 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.312590 22394 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.313503 22344 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.315493 22344 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.308015 22390 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:02.316210 22390 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:02.317131 22390 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.308105 22392 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.319788 21281 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:02.299456 22415 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.319973 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.303840 22374 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.303444 22422 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:02.279207 22402 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:02.324374 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 65599.5us
I20251214 02:59:02.324775 21248 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.275307 22419 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.326982 21098 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:02.328735 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.330595 22422 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.331030 22422 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:02.315950 22418 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.334465 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.315522 22416 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.334903 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.308398 22388 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.344241 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.327963 22415 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.328940 22392 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.329350 22407 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.349215 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.328197 22402 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.352284 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.353695 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.354398 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.331657 22418 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.359162 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.359309 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.363915 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.364040 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.364552 21124 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:02.365221 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.366029 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.366923 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.368808 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:02.369648 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.348325 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.369179 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.350783 22415 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.370434 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.371848 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.355096 22407 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.386422 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.398890 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.357695 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.400835 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.359781 22418 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.399734 22407 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.401654 22424 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.401870 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:02.382961 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.402068 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.408413 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:02.385375 22415 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.407836 22019 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.416230 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.278541 22420 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.408825 22389 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.432758 22012 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:02.402200 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.412369 22017 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:02.407765 22426 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.418000 22396 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.402098 22425 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.448496 22019 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:02.383105 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.453034 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.453286 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.450693 22380 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20251214 02:59:02.455278 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.456588 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.456967 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.458607 22020 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.460605 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.461871 21173 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:02.465255 21245 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.473837 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.454418 22425 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.464754 22396 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.440522 22389 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.480978 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.471526 22376 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.474411 22380 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:02.472467 22416 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:02.433632 22410 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:02.475313 22428 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.426199 22424 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.479313 22344 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:02.486863 21099 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.66:44049 timed out after 1.704s (SENT) [suppressed 1 similar messages]
I20251214 02:59:02.479497 22396 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.493822 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.454445 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:02.483188 22380 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:02.499028 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:02.499665 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:02.484884 22416 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:02.497578 21279 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.490341 22377 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:02.510250 22345 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.497040 22344 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.510804 22345 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.511236 22380 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.499313 22396 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.512476 22344 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.519893 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.521893 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.522265 22345 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.510799 22377 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.522709 22417 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:02.505152 22416 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.466106 22426 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.523782 21206 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:02.524250 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.480759 22374 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.512442 22380 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:02.508436 22011 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.528503 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.577163 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.557735 21208 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.558005 22374 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.505406 22427 consensus_queue.cc:1059] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.148s
I20251214 02:59:02.557778 22425 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.578804 22431 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20251214 02:59:02.580157 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.580236 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.576435 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.600319 21248 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.601444 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.523253 22417 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:02.490262 22419 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.554659 22016 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:02.586786 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 99954.3us
I20251214 02:59:02.500639 22428 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.607033 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.587458 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.592969 22389 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.593359 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.605000 21248 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.611315 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.608422 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.609408 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:02.590865 22420 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.603488 22398 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.621083 22392 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.603767 22425 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:02.606622 22345 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.625837 22425 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.607975 22424 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.608953 22430 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.637889 22425 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.638661 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.641434 22415 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.644956 21203 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:02.642125 22422 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.648617 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.649139 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.650882 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.651971 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.653383 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.654942 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.659804 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.644888 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.661432 22373 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
W20251214 02:59:02.674845 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 344553us
I20251214 02:59:02.669749 22373 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:02.676574 22373 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.646371 22407 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.678097 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:02.644727 21985 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.679059 22373 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.680060 22428 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.683894 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:02.668815 22382 raft_consensus.cc:3055] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
W20251214 02:59:02.685585 22434 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20251214 02:59:02.691154 22340 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:02.691545 22340 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:02.694057 22340 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.695188 22340 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.695607 22394 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.684837 22431 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.702987 22422 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.703970 22426 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:02.705600 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 96848.7us
I20251214 02:59:02.709235 22402 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.711314 22407 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.689726 22427 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.686086 22382 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:02.712127 22396 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.713100 22411 raft_consensus.cc:1064] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:02.715785 21028 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:02.698915 22428 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.723031 22415 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.753576 22411 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.723719 22427 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:02.758873 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 95836.8us
I20251214 02:59:02.720769 22388 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.758003 22427 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.724498 22417 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.780747 22020 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:59:02.759083 21985 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.778765 22429 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20251214 02:59:02.784468 21985 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.763623 22382 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.790635 22382 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:02.792322 22011 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:02.793485 22428 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.793905 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:02.787057 22426 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.794817 22361 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:02.810089 22402 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.786458 22427 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.812107 22427 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.815811 22418 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.794792 22014 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:02.822955 22396 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:02.823518 22361 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:02.824440 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.824460 22361 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.797435 22394 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:02.828437 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 348178us
I20251214 02:59:02.829185 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.831207 22396 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.831807 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.834285 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.789395 22430 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
W20251214 02:59:02.859799 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 254874us
I20251214 02:59:02.860797 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.858412 22418 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.830658 22374 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20251214 02:59:02.864955 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.866159 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.869731 21248 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.871191 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.872560 21248 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.863744 22418 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:02.879597 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.869166 22394 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.874951 22345 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.882789 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.872862 22430 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.886283 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.874310 22415 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.880517 22417 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.885331 22422 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.887310 22388 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:02.887619 22344 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.885597 22418 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:02.874791 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:02.901959 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 217551us
I20251214 02:59:02.903466 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.907552 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.907333 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:02.910315 22437 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20251214 02:59:02.913018 21248 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.904951 22392 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.914204 22392 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.914156 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.907883 22430 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.920298 22416 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:02.930246 22407 raft_consensus.cc:3055] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:02.929116 21031 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:02.930683 22407 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:02.934816 22416 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:02.935314 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:02.912184 22426 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.936162 22418 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:02.918804 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:02.920390 22394 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251214 02:59:02.913142 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 44418.4us
I20251214 02:59:02.943089 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.915831 22344 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.944396 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.945544 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.945578 21029 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:02.927949 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:02.954269 22344 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.956457 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.958145 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.960388 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.962579 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.963559 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:02.963972 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63618us
I20251214 02:59:02.965287 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.965687 22401 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:02.966735 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.965565 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.970124 21279 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:02.973018 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.973008 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:02.974884 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:02.975937 22433 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.976217 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:02.977677 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 59020.6us
I20251214 02:59:02.955750 22382 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:02.955447 22428 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.978579 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.978924 22411 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.960803 22426 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.980180 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.984328 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.985845 21098 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:02.992036 21098 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:02.992463 21201 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:59:02.993388 22376 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:02.993918 22380 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:02.994305 22380 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.994656 22380 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:02.994038 22396 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:02.996940 22344 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:03.000574 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:02.999728 21098 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:02.983023 22382 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:02.998878 22013 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:02.986310 22411 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:03.005191 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.017843 21098 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.003495 22428 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.022140 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.006707 22396 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.026402 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.007390 22382 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:03.030040 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002c67e0 after lost signal to thread 19984
I20251214 02:59:03.028621 22390 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.027709 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:03.029381 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.052907 22391 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.017482 22429 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:03.031037 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002da9e0 after lost signal to thread 21065
I20251214 02:59:03.011902 22411 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.021502 22399 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:02.981706 22407 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:03.059293 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:03.062577 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.028647 22427 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.063241 22423 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.066556 22011 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20251214 02:59:03.068375 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08003238c0 after lost signal to thread 21157
W20251214 02:59:03.069525 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002feb60 after lost signal to thread 21232
I20251214 02:59:03.075376 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.076995 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:03.079617 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 51991.7us
I20251214 02:59:03.080444 21098 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:03.071678 22041 debug-util.cc:398] Leaking SignalData structure 0x7b0800339f40 after lost signal to thread 21306
I20251214 02:59:03.081252 22429 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:03.088692 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.089424 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.089592 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.081122 22431 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:03.091790 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.092020 21273 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:03.092473 21282 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.089983 21275 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.083166 22325 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.097074 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.100947 21280 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.099409 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.101452 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.101696 21279 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.084107 22423 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:03.100581 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.089860 22429 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:03.107460 22041 debug-util.cc:398] Leaking SignalData structure 0x7b0800339740 after lost signal to thread 22044
I20251214 02:59:03.107806 22429 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.062721 21985 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.109263 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.094856 22394 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.094818 22391 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:03.100898 22410 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:03.098968 22407 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.117707 21275 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:03.111323 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.122159 21203 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:03.127751 21173 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.089550 22396 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20251214 02:59:03.128793 21206 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.104681 22447 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251214 02:59:03.123399 22444 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20251214 02:59:03.110808 22390 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:03.133090 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.110683 22437 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.126130 22428 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.148025 21208 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.132263 22325 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.138806 22410 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.143168 22399 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:03.139777 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 112886us
I20251214 02:59:03.178781 22390 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.205482 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.206214 21127 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:03.198251 22391 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251214 02:59:03.210685 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 68403.8us
I20251214 02:59:03.212163 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.221374 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:03.216838 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 54086.6us
I20251214 02:59:03.225034 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.204627 22439 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.227227 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.227861 21130 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:03.219756 22391 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:03.203433 22382 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.235587 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.239032 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.210068 21985 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.232682 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.235363 22391 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.256706 21280 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:03.256814 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.258209 21247 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.259238 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:03.259784 22453 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20251214 02:59:03.261073 21273 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.264374 22416 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.265504 22344 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.266117 22344 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:03.267160 22430 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.268141 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.270709 22416 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.272568 21247 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.273602 21279 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:03.273983 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.277706 22418 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.278972 22344 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.282861 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.235961 22410 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.246176 22390 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:03.261993 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 61242.8us
I20251214 02:59:03.268608 22391 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.288467 21985 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.289831 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.290557 22433 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.291222 22433 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.291492 22382 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.292428 21985 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.293574 22391 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.261116 22428 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.296166 22014 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.314049 22411 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.338673 22411 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.339084 22011 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:03.340727 22344 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:03.342505 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:03.343070 22380 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.344961 22380 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.385335 22429 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.403245 22429 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.415479 22426 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:03.421129 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 171196us
W20251214 02:59:03.422727 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 166573us
W20251214 02:59:03.423629 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 148350us
I20251214 02:59:03.424388 21098 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.427474 22376 consensus_queue.cc:1059] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:03.429184 22407 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:03.430032 22457 raft_consensus.cc:3055] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:03.430476 22457 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.428563 22376 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.431308 22457 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:03.432951 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 145318us
I20251214 02:59:03.434331 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.435653 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:03.450093 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 317079us
I20251214 02:59:03.451624 21173 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.453481 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.461983 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.471318 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.475162 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.476148 21173 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.459017 22429 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.476485 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.478014 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.480923 21277 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.482188 21280 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.479031 21124 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:03.479569 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:03.468092 22437 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.462779 22460 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.490573 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.487983 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.491986 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.501355 21128 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:03.502929 21133 raft_consensus.cc:1240] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:03.476614 22447 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.505923 21281 raft_consensus.cc:1240] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.479907 22455 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:03.501255 21279 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.511180 21129 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.486749 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.512678 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.478343 22401 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:03.512774 21127 raft_consensus.cc:1240] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.478165 22456 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:03.515574 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45110.9us
I20251214 02:59:03.507126 22377 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:03.522001 21098 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:03.525676 21130 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:03.519627 22373 consensus_queue.cc:1059] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20251214 02:59:03.526650 21125 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.508234 22361 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20251214 02:59:03.490794 22449 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20251214 02:59:03.530467 21207 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.518018 22455 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.548242 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.548532 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.549005 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.550232 21127 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:03.552915 21275 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.557274 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:03.558575 21282 raft_consensus.cc:1240] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.561617 21127 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:59:03.564393 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 45411.2us
I20251214 02:59:03.569571 21127 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.570158 21281 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:03.519687 22396 consensus_queue.cc:1059] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20251214 02:59:03.568174 21124 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.563373 22380 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:03.520244 22456 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:03.572952 21132 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.577474 22373 raft_consensus.cc:3055] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:03.566545 22377 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.543779 21985 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.588109 22411 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:03.591127 22018 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:03.591272 22455 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.592615 21130 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:03.594615 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:03.595376 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:03.565573 22458 consensus_queue.cc:1059] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251214 02:59:03.600250 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:03.571446 22430 consensus_queue.cc:1059] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20251214 02:59:03.621868 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 99207.3us
I20251214 02:59:03.622702 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:03.623298 22380 raft_consensus.cc:697] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.624464 22456 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:03.625013 22456 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:03.625461 22456 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:03.627002 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 46266.1us
I20251214 02:59:03.590171 22402 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251214 02:59:03.579962 22373 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.618708 22423 consensus_queue.cc:1059] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251214 02:59:03.624183 22380 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.632423 22376 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:03.637951 22373 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.639686 22373 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.620810 22426 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:03.637943 22423 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:03.641871 22423 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.613863 22411 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.642331 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:03.642886 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:03.642659 22423 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:03.592823 22377 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:03.644193 22411 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.618350 21173 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.636952 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:03.669759 22469 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20251214 02:59:03.676774 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 45001.4us
I20251214 02:59:03.674140 22389 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20251214 02:59:03.682142 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63876.4us
I20251214 02:59:03.630190 22430 raft_consensus.cc:3055] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:03.684592 22435 consensus_queue.cc:1059] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:03.631314 22428 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:03.678473 22418 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:03.665759 22423 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.631356 22437 consensus_queue.cc:1059] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20251214 02:59:03.694309 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:03.700408 21204 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:59:03.702818 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 152668us
I20251214 02:59:03.686533 22430 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:03.629607 22447 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20251214 02:59:03.708973 21247 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.666797 22376 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:03.704536 21205 raft_consensus.cc:2955] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:03.711030 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:03.686048 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63437.3us
I20251214 02:59:03.692759 22428 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:03.702709 22459 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.692869 22340 consensus_queue.cc:1059] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251214 02:59:03.734862 22376 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.741963 22376 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.747916 22472 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.748670 22472 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.749274 22472 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:03.703203 22390 consensus_queue.cc:1059] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251214 02:59:03.752301 22457 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:03.731168 22428 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.715950 22430 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.742081 22468 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:03.710717 22418 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.733074 22396 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:03.762082 22430 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.760675 22456 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.767400 22468 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 02:59:03.777220 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:03.791224 22389 raft_consensus.cc:3055] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:03.799793 22389 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.820253 22389 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.801476 22468 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.837491 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:03.825709 22389 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:03.838249 21204 raft_consensus.cc:2393] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:59:03.839274 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 161665us
I20251214 02:59:03.843940 21279 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
W20251214 02:59:03.846849 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 134805us
W20251214 02:59:03.853303 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 159092us
I20251214 02:59:03.853976 21131 raft_consensus.cc:1240] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:03.859325 22461 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:03.859681 22461 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.860975 22461 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.863324 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:03.863621 22401 consensus_queue.cc:1059] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:03.864205 22391 consensus_queue.cc:1059] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:03.864588 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:03.864933 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:03.865187 21274 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:03.865602 22391 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:03.866817 22391 raft_consensus.cc:3055] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:03.867154 22423 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:03.867303 22399 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.867561 22423 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.868369 22423 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.869515 22423 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.870648 22449 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:03.871534 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:03.871110 22449 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.872684 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:03.873768 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:03.876296 22449 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:03.862141 22461 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.864601 22401 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20251214 02:59:03.868920 22391 raft_consensus.cc:740] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.852267 22415 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:03.838222 22418 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:03.897161 21098 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:03.924113 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45010.4us
I20251214 02:59:03.923741 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:03.930675 21124 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:03.930935 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:03.932617 21279 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:03.883771 22391 consensus_queue.cc:260] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:03.937073 22410 raft_consensus.cc:3055] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:03.945892 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.948004 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:03.939261 21125 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:03.940866 22410 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:03.970539 22391 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:03.945540 22380 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20251214 02:59:03.998255 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49416.6us
I20251214 02:59:03.998822 21099 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:03.974058 22457 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:03.910957 22474 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
W20251214 02:59:04.019991 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 69975.5us
I20251214 02:59:04.020787 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:04.027424 22451 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:04.027915 22451 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:04.028318 22451 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:04.028604 22457 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:04.030005 21099 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:04.030323 22457 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:04.039330 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:04.047308 22478 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:04.056761 22407 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:04.057279 22407 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:04.057600 22407 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:04.058719 22463 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251214 02:59:04.033517 22449 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:04.077060 22483 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251214 02:59:04.079931 22475 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20251214 02:59:04.082890 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 203) took 1411 ms. Trace:
I20251214 02:59:04.083379 21028 rpcz_store.cc:276] 1214 02:59:02.671231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:02.671355 (+ 124us) service_pool.cc:225] Handling call
1214 02:59:04.082832 (+1411477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:03.014258 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:03.014947 (+ 689us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:03.014983 (+ 36us) write_op.cc:435] Acquired schema lock
1214 02:59:03.015002 (+ 19us) tablet.cc:660] Decoding operations
1214 02:59:03.015333 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:03.015412 (+ 79us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:03.015447 (+ 35us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:03.015640 (+ 193us) tablet.cc:699] Row locks acquired
1214 02:59:03.015661 (+ 21us) write_op.cc:260] PREPARE: finished
1214 02:59:03.015809 (+ 148us) write_op.cc:270] Start()
1214 02:59:03.015914 (+ 105us) write_op.cc:276] Timestamp: P: 1765681143015761 usec, L: 0
1214 02:59:03.015935 (+ 21us) op_driver.cc:348] REPLICATION: starting
1214 02:59:03.016614 (+ 679us) log.cc:844] Serialized 535 byte log entry
1214 02:59:03.544196 (+527582us) op_driver.cc:464] REPLICATION: finished
1214 02:59:04.056579 (+512383us) write_op.cc:301] APPLY: starting
1214 02:59:04.056660 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:04.056747 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:04.056865 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:04.057102 (+ 237us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:04.057255 (+ 153us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:04.057292 (+ 37us) write_op.cc:312] APPLY: finished
1214 02:59:04.057482 (+ 190us) log.cc:844] Serialized 22 byte log entry
1214 02:59:04.057646 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.057720 (+ 74us) write_op.cc:454] Released schema lock
1214 02:59:04.057757 (+ 37us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:04.058083 (+ 326us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.058108 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":141,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":499656,"apply.run_cpu_time_us":1487,"apply.run_wall_time_us":1486,"num_ops":1,"prepare.queue_time_us":264687,"prepare.run_cpu_time_us":34345,"prepare.run_wall_time_us":104363,"raft.queue_time_us":873979,"raft.run_cpu_time_us":48234,"raft.run_wall_time_us":475731,"replication_time_us":505252,"thread_start_us":808964,"threads_started":4,"wal-append.queue_time_us":111644}]]}
I20251214 02:59:04.034977 22410 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.083196 22428 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:04.087700 22410 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:04.089324 22428 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:04.093393 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:04.106254 22457 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:04.099963 22451 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.111924 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:04.119521 22464 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.546s user 0.004s sys 0.000s
I20251214 02:59:04.119807 22451 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:04.122217 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:04.139072 21124 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.139190 21127 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.146869 21273 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:04.146869 21277 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:04.157295 21282 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
W20251214 02:59:04.166860 22394 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.004s sys 0.005s
W20251214 02:59:04.167910 22394 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.004s sys 0.006s
W20251214 02:59:04.170696 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:04.133478 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:04.176898 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:04.186538 22428 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:04.179155 22460 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20251214 02:59:04.191452 22460 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:04.192586 21124 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.193986 22399 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:04.194392 22399 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:04.198871 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:04.192425 22460 consensus_queue.cc:1059] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:04.200668 22380 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:04.171140 22389 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251214 02:59:04.168609 22407 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.195197 22399 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:04.185498 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 59267.6us
I20251214 02:59:04.231062 22468 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:04.228994 22361 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20251214 02:59:04.208351 22460 raft_consensus.cc:3055] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:04.243392 21099 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:04.248648 21031 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 208) took 1359 ms. Trace:
I20251214 02:59:04.249029 21126 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
W20251214 02:59:04.251895 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 211732us
I20251214 02:59:04.255453 21277 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:04.257457 21280 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:04.254397 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:04.259135 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:04.260315 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:04.260850 21280 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:04.266680 21280 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:04.248400 22460 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:04.249024 21031 rpcz_store.cc:276] 1214 02:59:02.897756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:02.901680 (+ 3924us) service_pool.cc:225] Handling call
1214 02:59:04.248588 (+1346908us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:03.118820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:03.119264 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:03.122429 (+ 3165us) write_op.cc:435] Acquired schema lock
1214 02:59:03.126845 (+ 4416us) tablet.cc:660] Decoding operations
1214 02:59:03.194652 (+ 67807us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:03.195778 (+ 1126us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:03.196682 (+ 904us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:03.207451 (+ 10769us) tablet.cc:699] Row locks acquired
1214 02:59:03.211494 (+ 4043us) write_op.cc:260] PREPARE: finished
1214 02:59:03.224718 (+ 13224us) write_op.cc:270] Start()
1214 02:59:03.230236 (+ 5518us) write_op.cc:276] Timestamp: P: 1765681143223998 usec, L: 0
1214 02:59:03.231550 (+ 1314us) op_driver.cc:348] REPLICATION: starting
1214 02:59:03.268673 (+ 37123us) log.cc:844] Serialized 535 byte log entry
1214 02:59:03.854077 (+585404us) op_driver.cc:464] REPLICATION: finished
1214 02:59:04.058444 (+204367us) write_op.cc:301] APPLY: starting
1214 02:59:04.058511 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:04.061848 (+ 3337us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:04.070556 (+ 8708us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:04.079797 (+ 9241us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:04.079959 (+ 162us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:04.080006 (+ 47us) write_op.cc:312] APPLY: finished
1214 02:59:04.080230 (+ 224us) log.cc:844] Serialized 22 byte log entry
1214 02:59:04.080415 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.080492 (+ 77us) write_op.cc:454] Released schema lock
1214 02:59:04.080534 (+ 42us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:04.080868 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.080892 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6807,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":204177,"apply.run_cpu_time_us":1736,"apply.run_wall_time_us":22371,"mutex_wait_us":10264,"num_ops":1,"prepare.queue_time_us":183189,"prepare.run_cpu_time_us":3371,"prepare.run_wall_time_us":150125,"replication_time_us":622385,"thread_start_us":138275,"threads_started":1}]]}
I20251214 02:59:04.251976 22435 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:04.267182 22399 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:04.301265 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 58447.8us
I20251214 02:59:04.314313 21124 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.319906 22488 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:04.320425 22488 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:04.301179 22457 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251214 02:59:04.320070 22484 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20251214 02:59:04.329484 22447 raft_consensus.cc:3055] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:04.329890 22447 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:04.330847 22427 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:04.331230 22427 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:04.334913 22429 raft_consensus.cc:3055] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:04.335290 22429 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:04.335969 22429 consensus_queue.cc:260] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.336928 22429 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:04.332005 22391 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:04.339005 22391 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:04.339766 22391 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.322075 22488 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:04.336529 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 96) took 1393 ms. Trace:
W20251214 02:59:04.352298 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 52899.5us
I20251214 02:59:04.355235 21273 raft_consensus.cc:1240] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:04.355414 22460 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.356640 22460 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:04.344707 21029 rpcz_store.cc:276] 1214 02:59:02.943071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:02.943208 (+ 137us) service_pool.cc:225] Handling call
1214 02:59:04.336470 (+1393262us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:03.269096 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:03.269615 (+ 519us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:03.269647 (+ 32us) write_op.cc:435] Acquired schema lock
1214 02:59:03.269661 (+ 14us) tablet.cc:660] Decoding operations
1214 02:59:03.269984 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:03.270037 (+ 53us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:03.270064 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:03.270230 (+ 166us) tablet.cc:699] Row locks acquired
1214 02:59:03.270244 (+ 14us) write_op.cc:260] PREPARE: finished
1214 02:59:03.270370 (+ 126us) write_op.cc:270] Start()
1214 02:59:03.270486 (+ 116us) write_op.cc:276] Timestamp: P: 1765681143270324 usec, L: 0
1214 02:59:03.270507 (+ 21us) op_driver.cc:348] REPLICATION: starting
1214 02:59:03.282827 (+ 12320us) log.cc:844] Serialized 598 byte log entry
1214 02:59:04.014064 (+731237us) op_driver.cc:464] REPLICATION: finished
1214 02:59:04.126260 (+112196us) write_op.cc:301] APPLY: starting
1214 02:59:04.133624 (+ 7364us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:04.139188 (+ 5564us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:04.169015 (+ 29827us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:04.171945 (+ 2930us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:04.172098 (+ 153us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:04.172161 (+ 63us) write_op.cc:312] APPLY: finished
1214 02:59:04.172368 (+ 207us) log.cc:844] Serialized 22 byte log entry
1214 02:59:04.172589 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.172678 (+ 89us) write_op.cc:454] Released schema lock
1214 02:59:04.172717 (+ 39us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:04.173057 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:04.173082 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":69733,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":71560,"apply.run_cpu_time_us":1942,"apply.run_wall_time_us":48743,"num_ops":1,"prepare.queue_time_us":320415,"prepare.run_cpu_time_us":2334,"prepare.run_wall_time_us":13945,"replication_time_us":739734,"thread_start_us":149642,"threads_started":1}]]}
I20251214 02:59:04.368928 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:04.377338 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:04.379099 22505 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.067s user 0.003s sys 0.000s
I20251214 02:59:04.366750 22483 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:04.379652 22500 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251214 02:59:04.390364 21276 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:04.392439 22484 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:04.368098 22427 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.393929 22437 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:04.387384 22402 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:04.399176 21274 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:04.405375 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:04.413091 21203 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:04.414721 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:04.416806 21202 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:04.424754 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:04.383136 22451 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:04.428318 21208 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:04.402516 22437 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:04.414980 22407 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:04.447369 22018 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.401196 22483 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:04.448936 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:04.451658 22437 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.452678 22437 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:04.442786 22451 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:04.454710 22495 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:04.454519 22451 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:04.448753 22503 log.cc:927] Time spent T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
W20251214 02:59:04.468060 22507 consensus_queue.cc:1175] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e36caefa37d47a5968cffd9e270044f" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20251214 02:59:04.475320 22402 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:04.448428 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:04.487723 22501 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20251214 02:59:04.498792 22494 log.cc:927] Time spent T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
W20251214 02:59:04.502125 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:04.528681 22435 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:04.531016 22483 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:04.535718 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 62815us
I20251214 02:59:04.535960 22488 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.537544 22374 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:04.544988 22481 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:04.545555 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:04.550904 22013 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:04.589159 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 43701.6us
I20251214 02:59:04.593464 21277 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:04.596033 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:04.595134 21276 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:04.600733 22429 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:04.609730 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:04.673544 22401 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:04.684131 22401 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.686049 22401 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:04.794746 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:04.707690 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:04.971407 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 300094us
I20251214 02:59:04.681192 22396 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:04.975018 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:04.671484 22500 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.306s
I20251214 02:59:04.694351 22506 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:04.958861 22473 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:04.681684 22399 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:04.980576 22506 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.984006 22373 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251214 02:59:04.986873 22506 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:04.987063 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:04.984345 22447 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.988518 22447 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:04.990937 22374 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:04.991401 22374 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:04.993245 22374 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:04.994158 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:04.994776 21133 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:04.876806 22416 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:04.771796 22488 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.009382 22416 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:04.794746 22415 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.047645 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:05.048352 22483 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.048627 22488 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.008134 22399 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.040706 22435 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.048874 22483 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.049538 22435 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.051671 22474 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.051901 22435 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:04.979745 22396 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:05.056047 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:05.056605 21133 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:05.007170 22473 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.057904 22488 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:05.062294 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 86192.4us
I20251214 02:59:05.064513 21206 raft_consensus.cc:2393] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:05.069656 21133 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:05.071233 21282 raft_consensus.cc:2393] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:05.072372 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:05.073531 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:05.072981 21128 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:05.052176 22474 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.077071 21279 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:05.111385 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:05.047269 22415 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.067713 22483 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:05.174621 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:05.175010 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 62316.9us
I20251214 02:59:05.176012 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:05.213274 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.171514 22497 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:05.116838 22495 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.116952 22500 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:05.171648 22377 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.369092 21099 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:05.293898 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 121584us
W20251214 02:59:05.431906 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:05.211251 22519 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.211205 22390 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.459612 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:05.460022 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:05.460242 21277 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:05.460873 21128 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:05.238382 22340 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:05.350236 22447 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:05.462738 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:05.277652 22483 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:05.463711 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 272864us
I20251214 02:59:05.295024 22497 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.437774 22435 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.468968 21126 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:05.349783 22014 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:05.467247 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.375167 22416 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.240891 22526 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.368788 22474 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.438990 22407 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:05.461963 22511 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.409s user 0.001s sys 0.000s
I20251214 02:59:05.443085 22495 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.455639 22399 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.454488 22509 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.495127 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:05.439703 22500 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.468730 22447 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.461815 22340 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:05.467247 22459 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.482896 22396 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.479828 22497 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:05.440064 22377 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.497177 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.459542 22390 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.503003 22506 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.625890 22534 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.542582 22474 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.625267 22516 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.505833 22528 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.505901 22416 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.597343 22447 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:05.730326 21276 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:05.603585 22473 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.494933 22483 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.496047 22435 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.463901 22519 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.725464 22488 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.729200 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:05.743963 21279 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:05.738525 22500 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.747754 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:05.747829 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:05.639609 22399 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.732692 22415 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.763674 21274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 1757) took 1326 ms. Trace:
W20251214 02:59:05.764546 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:05.764961 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:05.764070 21274 rpcz_store.cc:276] 1214 02:59:04.436765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.436883 (+ 118us) service_pool.cc:225] Handling call
1214 02:59:04.448646 (+ 11763us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:04.449804 (+ 1158us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:04.449834 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:04.468900 (+ 19066us) log.cc:844] Serialized 26 byte log entry
1214 02:59:04.696049 (+227149us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:04.696085 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:04.696147 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:05.763462 (+1067315us) raft_consensus.cc:1685] finished
1214 02:59:05.763485 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:05.763619 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5321,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":226826,"threads_started":1,"wal-append.queue_time_us":504658}
I20251214 02:59:05.595461 22509 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.663017 22506 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.740247 22528 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.781937 22528 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.726115 22396 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20251214 02:59:05.712941 22474 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.626735 22459 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.502071 22526 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.739758 22416 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.787626 22390 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.739452 22534 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.759179 22399 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:05.801018 22535 log.cc:927] Time spent T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251214 02:59:05.801484 22390 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.747942 22340 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.785220 22524 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.799676 21279 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:05.749533 22516 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.803207 22390 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.805311 22340 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:05.808633 22459 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.797528 22520 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.789033 22530 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.785562 22539 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.809836 22524 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.820744 21282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50516 (request call id 1170) took 1131 ms. Trace:
I20251214 02:59:05.821162 21282 rpcz_store.cc:276] 1214 02:59:04.689344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.689531 (+ 187us) service_pool.cc:225] Handling call
1214 02:59:05.820495 (+1130964us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b540012edd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:05.820649 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223717760}
I20251214 02:59:05.828701 22390 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.818789 22416 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.825703 22377 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.820924 22524 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:05.831722 22460 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.859074 22534 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.815728 22520 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.861358 22520 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:05.881129 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:05.854308 22483 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.820325 22455 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:05.855196 22377 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:05.888772 22377 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.891271 22377 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:05.891947 22474 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:05.892614 22399 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.816985 22530 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.893574 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1128) took 1351 ms. Trace:
I20251214 02:59:05.854038 22435 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:05.886746 22509 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.888085 22526 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:05.854524 22519 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.885602 22506 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.893954 21130 rpcz_store.cc:276] 1214 02:59:04.542336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.542445 (+ 109us) service_pool.cc:225] Handling call
1214 02:59:05.892865 (+1350420us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:05.893060 (+ 195us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:05.893246 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.893273 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:05.893314 (+ 41us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:05.893343 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:05.893396 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:05.893495 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7574272}
I20251214 02:59:05.819298 22539 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.872416 22460 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.883354 22516 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:05.902098 22447 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.880360 22497 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.908061 22530 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.914089 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 574) took 1646 ms. Trace:
I20251214 02:59:05.927803 21127 rpcz_store.cc:276] 1214 02:59:04.268110 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.268230 (+ 120us) service_pool.cc:225] Handling call
1214 02:59:04.298052 (+ 29822us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:04.315384 (+ 17332us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:04.315423 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:04.318139 (+ 2716us) log.cc:844] Serialized 26 byte log entry
1214 02:59:04.376685 (+ 58546us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:04.376726 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:04.376784 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:05.913253 (+1536469us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b2400104c18. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bd449a4 00007f670bd42aab 00007f670bdd19a9 00007f670bdd1635 00007f670bdd0b90 00007f670eb65828 00007f670eb65f88 00007f670eb3d738 00007f670eb38b0a 00007f670eb37f56
1214 02:59:05.913813 (+ 560us) raft_consensus.cc:1685] finished
1214 02:59:05.913833 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:05.914032 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16858,"raft.queue_time_us":13725,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":3228,"spinlock_wait_cycles":5046016,"thread_start_us":54713,"threads_started":1,"wal-append.queue_time_us":141303}
I20251214 02:59:05.896281 22495 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.930987 22539 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.927551 22371 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:05.932577 22371 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.931578 22016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44274 (request call id 202) took 1282 ms. Trace:
I20251214 02:59:05.926095 22537 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.933270 22519 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:05.932396 22539 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:05.937451 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.038s sys 0.055s
W20251214 02:59:05.939337 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.039s sys 0.055s
I20251214 02:59:05.939992 21206 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
W20251214 02:59:05.940376 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 533252us
I20251214 02:59:05.942369 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.943269 21202 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:05.937135 22456 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:05.939088 22519 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.945878 21202 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:05.953128 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.953907 21200 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:05.958528 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.967754 22521 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.968492 22497 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.968320 22521 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.970394 22521 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.971614 22543 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:05.972223 22543 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.973949 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:05.974690 21200 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:05.975013 22509 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.975538 22509 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.977231 22509 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.979583 22371 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:05.935424 22016 rpcz_store.cc:276] 1214 02:59:04.648842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.930718 (+1281876us) service_pool.cc:225] Handling call
1214 02:59:05.931038 (+ 320us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:05.931238 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.931268 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:05.931312 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:05.931346 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:05.931416 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:05.931523 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:05.935514 22520 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:05.941772 22539 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:05.974593 22325 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.036170 22547 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.029671 22325 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:05.953203 22456 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.030033 22377 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.093683 22377 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.112682 22410 raft_consensus.cc:3055] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:06.113145 22410 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:06.114629 22513 log.cc:927] Time spent T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 1.066s user 0.014s sys 0.031s
I20251214 02:59:06.116117 21125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 577) took 1682 ms. Trace:
I20251214 02:59:06.116485 21125 rpcz_store.cc:276] 1214 02:59:04.434157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.436931 (+ 2774us) service_pool.cc:225] Handling call
1214 02:59:04.470764 (+ 33833us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:04.471126 (+ 362us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:04.471159 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:04.471585 (+ 426us) log.cc:844] Serialized 26 byte log entry
1214 02:59:04.920330 (+448745us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:04.920711 (+ 381us) log.cc:844] Serialized 14 byte log entry
1214 02:59:04.985555 (+ 64844us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:04.985635 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:06.114068 (+1128433us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:06.115819 (+ 1751us) raft_consensus.cc:1685] finished
1214 02:59:06.115926 (+ 107us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.116060 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63614,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":66384,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":174,"spinlock_wait_cycles":20902016,"thread_start_us":448375,"threads_started":1,"wal-append.queue_time_us":576914}
I20251214 02:59:06.118891 22371 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.119436 22371 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.120214 22456 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.006703 22520 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.120615 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 193) took 2644 ms. Trace:
I20251214 02:59:06.091533 22506 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.100894 22497 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:06.146206 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.001s sys 0.024s
I20251214 02:59:05.951860 22537 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.100611 22516 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:06.160543 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.001s sys 0.024s
I20251214 02:59:06.141378 21030 rpcz_store.cc:276] 1214 02:59:03.476052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:03.499695 (+ 23643us) service_pool.cc:225] Handling call
1214 02:59:06.120437 (+2620742us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20251214 02:59:06.113940 22410 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.205580 22361 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.206231 22361 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.199458 22495 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.230211 22495 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.160151 22447 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.158986 22506 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.048523 22547 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.177608 22497 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:06.416169 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:06.248603 22509 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.425776 22509 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.320407 22377 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:06.401185 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:06.392493 22543 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:06.416429 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 389390us
I20251214 02:59:06.427644 22543 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.292232 22526 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.428198 22543 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.428570 22526 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.430241 22543 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.439715 22543 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.232486 22460 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.248404 21029 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 1 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:06.299471 22013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 254) took 1232 ms. Trace:
I20251214 02:59:06.296391 21204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1789) took 1023 ms. Trace:
I20251214 02:59:06.196662 22516 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.243809 22537 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.511304 21204 rpcz_store.cc:276] 1214 02:59:05.272728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.272905 (+ 177us) service_pool.cc:225] Handling call
1214 02:59:06.295673 (+1022768us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:06.295859 (+ 186us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:06.296059 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:06.296088 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:06.296131 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:06.296159 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.296214 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.296329 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9861504}
I20251214 02:59:06.445943 22539 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:06.445495 22427 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:06.448182 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 576) took 2015 ms. Trace:
I20251214 02:59:06.480582 22410 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:06.523981 21029 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:06.530889 21208 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s user 0.028s sys 0.045s
W20251214 02:59:06.531932 21208 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s user 0.028s sys 0.045s
I20251214 02:59:06.481559 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1791) took 1208 ms. Trace:
I20251214 02:59:06.490473 22391 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:06.494195 22519 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:06.430326 22526 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:06.497462 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.498618 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.498692 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.498824 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.500372 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:06.502408 21984 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.008s sys 0.032s
W20251214 02:59:06.538025 21984 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.009s sys 0.032s
W20251214 02:59:06.502862 21199 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.041s
W20251214 02:59:06.505571 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:06.502094 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:06.507830 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 92219.4us
I20251214 02:59:06.541564 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.537593 21124 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
W20251214 02:59:06.510633 21247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.013s sys 0.054s
W20251214 02:59:06.513105 21248 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.229s user 0.022s sys 0.021s
W20251214 02:59:06.512091 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 84250.7us
W20251214 02:59:06.543051 21248 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.258s user 0.022s sys 0.022s
W20251214 02:59:06.445322 22340 consensus_meta.cc:287] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.637s user 0.007s sys 0.003s
W20251214 02:59:06.526669 21986 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.021s sys 0.038s
I20251214 02:59:06.527200 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.505892 22431 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.495980 22423 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:06.542310 21247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.070s user 0.013s sys 0.055s
W20251214 02:59:06.539719 21199 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s user 0.000s sys 0.041s
I20251214 02:59:06.552990 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.551002 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.553781 21128 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:06.551021 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.554119 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.554718 21204 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:06.555505 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.532287 21205 rpcz_store.cc:276] 1214 02:59:05.273308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.722868 (+449560us) service_pool.cc:225] Handling call
1214 02:59:05.749481 (+ 26613us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:05.791290 (+ 41809us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.791319 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:05.791360 (+ 41us) raft_consensus.cc:1645] Marking committed up to 2
1214 02:59:05.791389 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.481307 (+689918us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b540008f554. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:06.481353 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.481492 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95713536}
I20251214 02:59:06.561602 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:06.562799 21986 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.021s sys 0.038s
I20251214 02:59:06.563086 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.563911 21128 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:06.530627 22539 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.568385 21199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1788) took 1321 ms. Trace:
I20251214 02:59:06.568573 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.563277 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.565099 21204 raft_consensus.cc:2393] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:06.565762 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.566820 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.568737 21199 rpcz_store.cc:276] 1214 02:59:05.247381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.247517 (+ 136us) service_pool.cc:225] Handling call
1214 02:59:05.906539 (+659022us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:05.906959 (+ 420us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:05.932749 (+ 25790us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.932779 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:05.933229 (+ 450us) log.cc:844] Serialized 26 byte log entry
1214 02:59:06.553408 (+620179us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:06.553445 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.553508 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:06.568141 (+ 14633us) raft_consensus.cc:1685] finished
1214 02:59:06.568160 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.568331 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25370,"raft.queue_time_us":25524,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":72400128,"thread_start_us":619823,"threads_started":1,"wal-append.queue_time_us":633189}
I20251214 02:59:06.567445 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.570730 21206 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:06.570932 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.569149 21200 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:06.570186 21126 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:06.572916 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.577113 22506 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.573554 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.577718 22516 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:06.574518 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.062s user 0.033s sys 0.062s
W20251214 02:59:06.578475 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.179s user 0.033s sys 0.062s
I20251214 02:59:06.578816 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.529307 21131 rpcz_store.cc:276] 1214 02:59:04.433216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.435948 (+ 2732us) service_pool.cc:225] Handling call
1214 02:59:06.391988 (+1956040us) spinlock_profiling.cc:243] Waited 7.54 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2e3e 000000000035a3c9 00007f6713edd560 00007f6713fa5dd4 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:06.444714 (+ 52726us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:06.446427 (+ 1713us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:06.446459 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:06.446509 (+ 50us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:06.446542 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.447981 (+ 1439us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.448118 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17333632}
I20251214 02:59:06.577070 21204 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:06.525853 22543 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.562096 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.580479 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.577898 21205 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:06.576231 22497 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.589267 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.589915 21204 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:06.590022 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.584977 21124 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:06.590667 21206 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:06.558488 22361 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:06.565011 22377 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.594902 21247 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.594151 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.596068 21200 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:06.572817 22555 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.601697 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.606588 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.564847 22423 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.585939 22519 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.587165 22380 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.618773 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:06.562369 22431 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.574666 22539 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:06.611984 21983 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.018s sys 0.014s
I20251214 02:59:06.623497 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.626247 22017 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:06.601365 22377 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.603014 22517 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.602878 22526 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.609268 22516 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:06.633284 22553 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.648392 22553 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.648721 22553 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:06.594539 22016 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:06.640470 22377 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:06.628041 21099 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.655849 21099 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.640901 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:06.626431 21983 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.019s sys 0.014s
I20251214 02:59:06.511186 22013 rpcz_store.cc:276] 1214 02:59:05.066840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.068442 (+ 1602us) service_pool.cc:225] Handling call
1214 02:59:06.295070 (+1226628us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800212a98. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:06.295293 (+ 223us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:06.298481 (+ 3188us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:06.299107 (+ 626us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:06.299158 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:06.299192 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.299255 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.299387 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2384640}
I20251214 02:59:06.644165 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.670769 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.671440 21202 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:06.673550 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.673751 21099 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.674175 21204 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:59:06.689558 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 48767.2us
I20251214 02:59:06.692680 21098 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:06.626540 22374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.034s sys 0.025s
I20251214 02:59:06.659703 22495 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.623975 22519 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.645609 22325 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.700191 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.643193 22371 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.662665 22544 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:06.701498 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.689785 22550 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.702359 22449 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.705988 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:47052 (request call id 818) took 2161 ms. Trace:
I20251214 02:59:06.675486 22523 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:06.709369 21099 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.629788 22380 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:06.759078 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 49804.2us
I20251214 02:59:06.620620 22555 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.711714 22537 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:06.725117 21200 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:06.767541 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:06.659325 22517 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.751691 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:06.752215 21124 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:06.658138 22546 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20251214 02:59:06.676177 22516 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:06.757061 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.665496 22556 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.658612 22526 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.687657 22554 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:06.709542 22374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.350s user 0.036s sys 0.027s
I20251214 02:59:06.882445 22555 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:06.883167 22559 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.062s user 0.002s sys 0.000s
I20251214 02:59:06.706425 22440 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:06.755906 21132 rpcz_store.cc:276] 1214 02:59:04.537234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.540507 (+ 3273us) service_pool.cc:225] Handling call
1214 02:59:05.933017 (+1392510us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5d46 00007f6713fa5f20 00007f6713f57472 00007f670d4f6437 00007f670d4f63c2 00007f670d4f62f1
1214 02:59:06.698034 (+765017us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54000daa54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6713f5758e 00007f670d4f6437 00007f670d4f63c2 00007f670d4f62f1 00007f670d4f626c 00007f670d4f5522 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:06.698595 (+ 561us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1742682880}
I20251214 02:59:06.709707 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.706096 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.753635 22544 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.773048 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.777365 21099 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.757123 22449 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.675704 22518 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.756660 22523 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.782110 22423 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:06.786949 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.895283 21281 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:06.895632 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.787652 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.787825 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1782) took 1595 ms. Trace:
I20251214 02:59:06.788304 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.790373 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.791054 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.898306 21280 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:06.794509 22020 raft_consensus.cc:2393] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:06.794432 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.761276 22520 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.796023 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.753180 22560 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.761929 22509 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.765641 22547 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.818943 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.758685 22550 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:06.902835 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 125571us
I20251214 02:59:06.897441 21208 rpcz_store.cc:276] 1214 02:59:05.191960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.211329 (+ 19369us) service_pool.cc:225] Handling call
1214 02:59:05.211699 (+ 370us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:05.469751 (+258052us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.469783 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:05.495733 (+ 25950us) log.cc:844] Serialized 26 byte log entry
1214 02:59:06.532287 (+1036554us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:06.532332 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.532405 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:06.787603 (+255198us) raft_consensus.cc:1685] finished
1214 02:59:06.787623 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.787770 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6099,"raft.queue_time_us":104301,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":19829,"thread_start_us":1195430,"threads_started":2,"wal-append.queue_time_us":1178834}
I20251214 02:59:06.904310 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.768406 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.780221 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.794597 22429 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.928519 21124 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:06.900087 21273 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:06.909875 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.930373 21208 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:06.908190 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:06.833415 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 121600us
I20251214 02:59:06.842461 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.933203 21128 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:06.933990 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.850543 22373 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.935141 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.832635 22556 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.941247 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.852084 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.852641 22516 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.853627 22517 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:06.840384 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:06.855468 22380 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.826313 22013 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:59:06.855695 22431 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.856230 22539 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.857079 22519 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:06.794411 22543 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:06.859436 21130 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:06.826690 22566 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.839637 22537 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.974614 22537 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.974727 22566 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.975888 22550 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:06.851191 22565 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:06.870878 22377 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.871654 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.872105 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:06.873584 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:06.844620 22541 log.cc:927] Time spent T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20251214 02:59:06.765836 22478 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.880910 22554 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:06.893963 21133 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:06.787271 21201 raft_consensus.cc:2410] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251214 02:59:06.896903 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.899340 21278 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:06.897760 21276 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:59:06.938392 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.893421 22440 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.897964 22449 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:06.901829 22544 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:06.899971 22523 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:06.899307 22553 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:06.917243 22495 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251214 02:59:06.949095 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:06.935914 22340 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:06.929217 22429 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:06.963990 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.906581 22567 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.944160 22509 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:06.969615 22577 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.970191 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:06.974004 21199 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:06.993474 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.008158 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.009469 21132 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:06.992400 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.013547 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:07.015969 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 77653.4us
I20251214 02:59:07.016014 21986 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:06.957134 22517 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:06.961112 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1129) took 2405 ms. Trace:
I20251214 02:59:07.018308 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:07.014756 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 50746.2us
I20251214 02:59:07.037668 21130 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:06.902917 22518 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.039170 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:06.971517 22483 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:06.976626 22566 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.040830 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.049757 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.044579 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.007653 22340 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.033900 22543 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:07.059350 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 187864us
I20251214 02:59:07.038527 22361 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.038051 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.007589 22495 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.047814 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.048311 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.050529 22410 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:07.048652 21096 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:07.008710 22478 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:06.935793 22560 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.049135 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.051076 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.071506 21205 raft_consensus.cc:2410] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251214 02:59:06.976833 22537 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.052533 21274 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:07.011425 22440 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.011409 22429 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.053184 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.980906 22556 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.012008 22509 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.018322 22397 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:07.054754 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.055166 21130 raft_consensus.cc:2368] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3.
I20251214 02:59:07.033851 22500 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.053171 22483 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.032167 22550 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:07.040042 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 94133.1us
I20251214 02:59:07.061072 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:06.945931 22373 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.055426 22340 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.065320 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.132973 21282 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:07.070225 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.071954 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.014405 22577 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:07.134107 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 62224.4us
I20251214 02:59:07.074437 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.017591 22567 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.079198 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.061832 22361 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.080988 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.038589 22517 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.064064 22519 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:07.082624 21203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1785) took 1869 ms. Trace:
I20251214 02:59:07.068037 22495 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.035624 22565 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.060326 21245 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:07.103077 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.129976 21275 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:07.140601 22547 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:07.134927 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.151839 21124 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:07.072129 22478 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.069479 22410 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.137670 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.103004 22575 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.076090 22391 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.138546 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.139966 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.102802 22500 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.145534 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.159198 21282 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:07.131491 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.142905 21203 rpcz_store.cc:276] 1214 02:59:05.213135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.213241 (+ 106us) service_pool.cc:225] Handling call
1214 02:59:05.213561 (+ 320us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:05.355216 (+141655us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:05.355249 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:05.454678 (+ 99429us) log.cc:844] Serialized 26 byte log entry
1214 02:59:05.980414 (+525736us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b48000b22d8. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6713eb7e21 00007f6713eaeeb7 00007f6713eaed4a 00007f6713fa5f7f 00007f6713fa5c47 00007f6713fba26c 00007f6713fba218 00007f6713fba198 00007f6713fba153 00007f6713fb93fc 00007f6712ef4516
1214 02:59:05.982776 (+ 2362us) raft_consensus.cc:1645] Marking committed up to 2
1214 02:59:05.982819 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:05.982884 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:07.067214 (+1084330us) raft_consensus.cc:1685] finished
1214 02:59:07.067232 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:07.082539 (+ 15307us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102582,"raft.run_cpu_time_us":21387,"raft.run_wall_time_us":629719,"spinlock_wait_cycles":2888448,"thread_start_us":626832,"threads_started":2,"wal-append.queue_time_us":1047686}
I20251214 02:59:07.149531 22565 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.115417 22509 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.084050 22411 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:07.136763 22458 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:07.155748 21273 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:07.156575 21276 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:07.144520 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.156766 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.164983 21173 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.158439 22391 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.149371 21278 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251214 02:59:07.142519 22361 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.139117 22572 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.166406 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:07.167411 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 118852us
I20251214 02:59:07.167827 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.170423 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.157933 22410 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:07.155458 22579 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:07.161716 22547 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:07.171756 21282 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:07.199136 22579 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.139909 22397 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.161140 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:07.201021 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 140388us
I20251214 02:59:07.161202 22423 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:07.105582 22526 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:07.175794 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 115562us
I20251214 02:59:07.185091 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:07.175623 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 44245.1us
I20251214 02:59:07.142212 22550 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.214841 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:07.179853 22573 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20251214 02:59:07.179373 22576 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20251214 02:59:07.168498 22391 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.170840 22520 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.176416 22377 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.150426 22519 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.162844 22500 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.172816 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.210595 21274 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:07.211210 22579 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.173899 22401 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.170375 22371 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:07.216105 22361 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.176162 22571 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.175979 22566 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:07.178783 22553 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.201472 21125 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:07.177891 22373 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.160236 22575 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:07.228790 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:07.262049 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 51974.9us
I20251214 02:59:07.227187 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.210587 21985 leader_election.cc:400] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3.
I20251214 02:59:07.209206 22423 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.233067 22550 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.036923 21129 rpcz_store.cc:276] 1214 02:59:04.542672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.542801 (+ 129us) service_pool.cc:225] Handling call
1214 02:59:06.463741 (+1920940us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:06.463999 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:06.464229 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 5
1214 02:59:06.674902 (+210673us) log.cc:844] Serialized 14 byte log entry
1214 02:59:06.893638 (+218736us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:06.900380 (+ 6742us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:06.902681 (+ 2301us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:06.941850 (+ 39169us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:06.948282 (+ 6432us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54944,"prepare.queue_time_us":247808,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":39549,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":5372,"spinlock_wait_cycles":24030592,"thread_start_us":317476,"threads_started":2,"wal-append.queue_time_us":278304}
I20251214 02:59:07.239180 22397 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.249789 22571 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.245116 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.281126 21205 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:07.225067 22520 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.258962 22481 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:07.254050 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.260855 22565 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.289588 22020 raft_consensus.cc:2393] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:07.171706 22458 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.260918 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:07.176851 22572 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.256165 22391 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.266419 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.266952 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.264724 22553 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.264614 21985 leader_election.cc:403] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:07.247046 22371 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:07.269938 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.245626 22401 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.225723 22544 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.276188 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.276710 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.278457 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.239480 22429 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.249548 22566 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.249357 22377 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:07.286993 22571 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.267140 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.541731 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.290815 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.507352 21277 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:07.507143 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:07.508824 21276 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:07.509615 21275 raft_consensus.cc:2410] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251214 02:59:07.511212 22556 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.510939 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:07.511683 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 349487us
W20251214 02:59:07.364135 22552 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251214 02:59:07.342300 22560 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.526592 21127 raft_consensus.cc:2368] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:07.528009 22544 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.508877 22371 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:07.563643 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.564006 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.564392 21124 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:07.535102 22566 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:07.269114 22423 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:07.530310 22391 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.506904 22554 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.506834 22586 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.530622 22572 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.359025 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.532238 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.338593 22570 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:07.532778 21173 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.974s (SENT) [suppressed 4 similar messages]
I20251214 02:59:07.532721 22461 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.533223 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:07.288247 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:07.533812 21201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 1769) took 3097 ms (client timeout 2208 ms). Trace:
I20251214 02:59:07.251421 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:07.435395 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 143842us
I20251214 02:59:07.539554 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.362392 21207 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251214 02:59:07.544379 22575 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.544790 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.547999 22587 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:07.548445 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:07.287438 22520 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.552120 21280 raft_consensus.cc:2368] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:07.551254 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.555307 21132 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:07.357261 22588 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.509374 22401 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.564570 21277 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:07.337968 22550 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:07.339020 22481 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.265404 22547 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.353431 22518 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.566511 22540 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:07.419284 22500 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.587337 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.339334 22565 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:07.589784 21207 pending_rounds.cc:85] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 1
I20251214 02:59:07.569479 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.545773 22553 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.571410 22567 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.611912 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.546346 22377 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.588065 22577 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.638803 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.608860 22461 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.640874 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:07.714277 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 181153us
I20251214 02:59:07.747350 21280 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
W20251214 02:59:07.714320 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 105690us
I20251214 02:59:07.611032 22587 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.588946 21129 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:07.752182 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.578380 22586 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.714099 21207 pending_rounds.cc:107] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251214 02:59:07.620250 22519 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:07.744240 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 211521us
I20251214 02:59:07.634663 22361 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:07.717463 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 106621us
I20251214 02:59:07.744777 21281 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:07.745468 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.786758 21132 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:07.787904 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.745574 21274 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:07.749763 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.718508 22374 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:07.778726 21207 raft_consensus.cc:2889] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251214 02:59:07.719013 22509 consensus_meta.cc:287] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.549s user 0.007s sys 0.000s
W20251214 02:59:07.796372 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47956us
I20251214 02:59:07.589553 22570 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.789783 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:07.588968 21201 rpcz_store.cc:269] 1214 02:59:04.410982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.416292 (+ 5310us) service_pool.cc:225] Handling call
1214 02:59:07.341981 (+2925689us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54000cb054. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:07.508173 (+166192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5278531456}
I20251214 02:59:07.791621 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.746495 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:07.794611 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.746395 22458 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.753365 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.643540 22588 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.800649 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.829347 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.829993 21132 raft_consensus.cc:2368] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:07.830225 21124 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:07.803191 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.810847 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:07.813266 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 65095.9us
I20251214 02:59:07.783768 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.816023 22517 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.817219 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.820766 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.821090 22570 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:07.841231 22449 consensus_meta.cc:287] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.822s user 0.007s sys 0.001s
I20251214 02:59:07.818331 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.713805 22540 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.641642 22550 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.822866 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.825676 22553 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.828179 22483 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.833635 21281 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:07.835232 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.835675 22481 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.836046 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:07.836283 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:07.809067 22556 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3.
I20251214 02:59:07.862825 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.836458 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:07.865275 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:07.866852 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:07.837132 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.837244 22587 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.835844 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:07.838809 22440 consensus_meta.cc:287] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.757s user 0.000s sys 0.009s
I20251214 02:59:07.839108 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.829144 22560 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:07.841423 22371 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.809605 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 1787) took 2581 ms. Trace:
I20251214 02:59:07.881598 21207 rpcz_store.cc:276] 1214 02:59:05.228218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:05.228342 (+ 124us) service_pool.cc:225] Handling call
1214 02:59:07.353705 (+2125363us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:07.792892 (+439187us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:07.809392 (+ 16500us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b54000cb050. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:07.809514 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4888304000}
I20251214 02:59:07.800208 22509 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.843710 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.844177 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:07.845261 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:07.810014 22596 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20251214 02:59:07.828751 22410 consensus_meta.cc:287] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.618s user 0.004s sys 0.004s
W20251214 02:59:07.855460 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:07.829403 22325 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:07.800320 22429 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.838744 22588 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.861493 21273 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.876677 22553 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:07.863799 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:07.872563 22478 consensus_meta.cc:287] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.710s user 0.006s sys 0.000s
I20251214 02:59:07.874789 22517 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.842326 22449 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.909202 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:07.876287 21248 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:07.843770 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:07.895655 22325 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.887153 22371 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.879026 22440 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.879143 22570 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.867727 21280 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:07.912868 22325 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:07.900111 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.903081 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.904878 21124 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:07.884188 22500 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.894249 22410 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.907440 21200 raft_consensus.cc:2393] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:07.879318 21129 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 1.
I20251214 02:59:07.887303 22540 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:07.896392 22340 consensus_meta.cc:287] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.753s user 0.006s sys 0.015s
I20251214 02:59:07.920110 22570 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:07.930233 22340 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.910845 21274 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.885697 21132 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:07.931952 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47078 (request call id 588) took 1051 ms. Trace:
I20251214 02:59:07.909847 21098 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:07.909471 22478 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.933583 22540 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:07.863200 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 02:59:07.934403 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 78478.1us
I20251214 02:59:07.932364 21131 raft_consensus.cc:2393] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:07.935521 22556 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.935340 22520 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.936111 22509 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.932255 21132 rpcz_store.cc:276] 1214 02:59:06.880288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:07.843493 (+963205us) service_pool.cc:225] Handling call
1214 02:59:07.931896 (+ 88403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:07.933842 22340 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.914407 22550 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.911019 21248 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:07.912703 22567 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:07.879624 22560 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.915202 22449 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.915741 21126 raft_consensus.cc:2368] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:07.906435 22373 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20251214 02:59:07.915202 22565 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.943030 22560 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.918870 22586 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:07.898227 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.921344 21206 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:07.906910 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.920464 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.938225 22440 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.928054 21130 raft_consensus.cc:2393] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:07.923780 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:07.929678 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.931409 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.920042 22461 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.949608 21125 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:07.935914 22556 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:07.908229 22517 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:07.955037 21128 raft_consensus.cc:2368] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:07.937283 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:07.936753 21205 raft_consensus.cc:2368] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
W20251214 02:59:07.914543 22495 consensus_meta.cc:287] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.766s user 0.008s sys 0.001s
I20251214 02:59:07.939945 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:07.958943 21132 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:07.944877 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.944977 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:07.878762 22571 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:07.946354 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:07.946738 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:07.946836 21099 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.962240 21208 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.950583 21203 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.952255 21273 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:07.943704 22550 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.932583 22570 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.966938 22423 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.968837 22423 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.937520 22520 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:07.917753 22401 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.959730 21248 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:07.959978 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.960893 22440 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:07.963027 21199 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.963116 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.961273 22517 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.963042 22586 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.964931 22550 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:07.965098 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.982417 21127 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.961004 21129 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:07.970928 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.990360 21199 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:07.961243 21098 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:07.978866 21282 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:07.962163 22495 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:07.979907 21132 raft_consensus.cc:2393] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:07.964046 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.992317 21204 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:07.992599 21099 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:07.980264 22377 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:07.961620 22556 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.977313 21201 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:07.981150 22461 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.952685 22011 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:07.969563 22340 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:07.985299 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:07.996168 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1167) took 1127 ms. Trace:
I20251214 02:59:07.987982 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:07.986718 22523 consensus_meta.cc:287] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.968s user 0.008s sys 0.002s
I20251214 02:59:07.998804 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:08.001710 21132 rpcz_store.cc:276] 1214 02:59:06.868964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:07.995607 (+1126643us) service_pool.cc:225] Handling call
1214 02:59:07.995861 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:07.995973 (+ 112us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:07.995988 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:07.996017 (+ 29us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:07.996034 (+ 17us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:07.996066 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:07.996129 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:08.002342 21206 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:59:07.986409 22550 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.007478 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:07.984485 22586 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:08.009701 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47062 (request call id 1165) took 1141 ms. Trace:
I20251214 02:59:08.010056 21128 rpcz_store.cc:276] 1214 02:59:06.868402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:06.933738 (+ 65336us) service_pool.cc:225] Handling call
1214 02:59:08.009507 (+1075769us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540032a354. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.009630 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2342708608}
I20251214 02:59:08.011875 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47062 (request call id 1169) took 1142 ms. Trace:
I20251214 02:59:08.012178 21130 rpcz_store.cc:276] 1214 02:59:06.869478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:07.134685 (+265207us) service_pool.cc:225] Handling call
1214 02:59:08.011704 (+877019us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b54000da554. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.011821 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1788437248}
I20251214 02:59:08.012900 21274 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:08.013779 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.014030 21099 leader_election.cc:400] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:08.015959 22565 raft_consensus.cc:2764] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251214 02:59:08.014225 21132 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:08.014329 21099 leader_election.cc:403] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.017735 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.017999 21126 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47062 (request call id 1161) took 1263 ms. Trace:
I20251214 02:59:08.018363 21126 rpcz_store.cc:276] 1214 02:59:06.755023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:06.755154 (+ 131us) service_pool.cc:225] Handling call
1214 02:59:08.017830 (+1262676us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002bf054. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.017944 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2639543552}
I20251214 02:59:07.994197 22440 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.020592 21173 leader_election.cc:400] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.021617 21173 leader_election.cc:403] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.034533 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:07.997643 22582 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.040961 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.041396 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.037003 21247 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:08.042605 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.037555 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.043279 21199 raft_consensus.cc:2368] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.043300 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.025131 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.022992 21173 leader_election.cc:400] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.044262 21173 leader_election.cc:403] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.044243 21127 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:08.045073 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.026805 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.038458 22572 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:07.998018 22461 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:08.009140 22340 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:08.049510 22575 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:08.049937 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.017234 22574 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:08.053042 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.055336 21247 leader_election.cc:400] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.067175 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.011153 22586 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.057425 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.058825 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.022603 22478 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.063449 21096 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.050290 22553 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.069197 21277 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:08.059454 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.077303 21279 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:08.068744 21247 leader_election.cc:400] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.043471 22410 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.054364 22572 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:08.010763 22523 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.065512 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.047927 22382 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:08.077972 21247 leader_election.cc:403] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.038726 22481 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:08.073959 22371 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.081779 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.082722 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.072113 22582 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.084090 21132 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:08.084100 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.063702 22340 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.088732 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.072369 22518 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.071949 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.065900 22575 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:08.069599 22574 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:08.092037 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.116654 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.093236 21273 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.094144 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.096266 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:08.096727 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.099548 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.100131 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.118791 21281 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.100670 21124 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:08.102963 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.119951 21279 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:08.104938 22495 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.105340 21203 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:08.051115 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47062 (request call id 1160) took 1342 ms. Trace:
I20251214 02:59:08.120860 21131 rpcz_store.cc:276] 1214 02:59:06.709150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:06.712287 (+ 3137us) service_pool.cc:225] Handling call
1214 02:59:08.022456 (+1310169us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54001f1dd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.047655 (+ 25199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2291731840}
I20251214 02:59:08.063866 22461 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.111341 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.113281 21133 raft_consensus.cc:2368] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.086194 22546 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.092078 22572 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.128763 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.128949 22567 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.129055 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.129810 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.130218 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.134121 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.134927 21248 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.125660 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.125801 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.127197 22410 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.130806 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.131481 22565 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.131847 22519 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:08.132808 21277 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.115492 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.116343 22575 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:08.137972 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.123696 22574 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.139990 22518 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.162801 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.133631 22523 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.142179 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:08.148204 22567 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.148423 21280 raft_consensus.cc:2376] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4. Re-sending same reply.
I20251214 02:59:08.143051 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.176616 21201 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:08.152782 22423 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:08.181591 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.181983 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.182288 21124 raft_consensus.cc:2368] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.184885 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.185693 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.178486 22423 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.141621 21248 leader_election.cc:400] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.151742 22591 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.133291 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.151130 21273 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:08.153544 21099 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.153905 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.155865 21126 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.140422 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.158288 21208 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:08.213824 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.123413 22571 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:08.135737 22566 consensus_meta.cc:287] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.559s user 0.002s sys 0.008s
I20251214 02:59:08.162269 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.141680 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.133328 22325 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.220492 21131 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:08.221341 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.221560 22325 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.109963 22481 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.222069 22382 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.221823 22574 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.222558 22382 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.210314 22517 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.223392 22382 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.224056 22481 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:08.165414 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.224788 22517 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.225430 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.226552 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.226913 22572 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.227557 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.228293 21282 raft_consensus.cc:2393] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:08.228991 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.229920 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.230376 22572 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:08.231683 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.169751 21129 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:08.241259 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.162252 22410 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:08.170749 22582 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.171651 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.139302 22371 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:08.138963 22560 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:08.162335 22602 log.cc:927] Time spent T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
W20251214 02:59:08.155303 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 93955.8us
I20251214 02:59:08.155157 22519 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:08.245283 22519 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.149125 22401 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.214461 22553 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.235664 22547 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:08.186692 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.148873 21205 raft_consensus.cc:2368] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.191325 21206 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:08.192142 21096 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:08.249826 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:08.249964 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 57920.2us
I20251214 02:59:08.193230 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.197289 21098 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.200742 22461 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.201637 22570 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.204083 21248 leader_election.cc:403] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.204061 21128 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:08.207139 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47078 (request call id 590) took 1304 ms. Trace:
I20251214 02:59:08.153221 21203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:08.256831 21203 raft_consensus.cc:606] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:08.256471 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 119086us
I20251214 02:59:08.257825 21248 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.258791 21248 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.257025 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.259804 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.122809 22556 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:08.214622 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 61172.6us
I20251214 02:59:08.263501 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.256588 21133 rpcz_store.cc:276] 1214 02:59:06.903207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:07.007905 (+104698us) service_pool.cc:225] Handling call
1214 02:59:08.206957 (+1199052us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002beb54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.207086 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2538920192}
I20251214 02:59:08.264597 21099 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.264930 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.215276 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:08.216061 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 62251.7us
I20251214 02:59:08.218593 22566 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.218590 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:08.219832 22423 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.235539 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.237727 21281 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:08.269654 21273 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:08.269914 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.238420 21207 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:08.243062 21276 raft_consensus.cc:2368] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:08.270185 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.148078 22340 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:08.253227 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56035.9us
I20251214 02:59:08.260452 21203 raft_consensus.cc:614] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:08.267056 21099 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.246161 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 97) took 3851 ms. Trace:
I20251214 02:59:08.246064 21125 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:08.248569 22547 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:08.247331 22371 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:08.275236 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.284592 21280 raft_consensus.cc:2376] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4. Re-sending same reply.
I20251214 02:59:08.275512 21248 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.249523 22520 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.277878 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.260979 22500 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.289444 21278 raft_consensus.cc:2368] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
I20251214 02:59:08.271723 22458 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.290381 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.273550 21281 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:08.281708 21277 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:08.267539 22517 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.284518 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.268859 22518 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.287181 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.257601 22565 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.279423 21132 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:08.287794 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.287806 21173 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.304709 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.305442 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.265427 22567 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.257707 22373 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.291143 21279 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:08.285928 22340 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.268304 22435 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:08.300638 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.305838 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.318805 21203 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.319640 21173 leader_election.cc:400] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.321038 21173 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.286741 22547 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.268739 21985 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.293398 22361 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.293867 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:08.314885 22571 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.304370 22500 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.312052 22592 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.316144 22587 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:08.335541 22495 raft_consensus.cc:2764] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251214 02:59:08.356578 22587 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:08.356987 22587 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:08.357613 22566 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.358291 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:08.332227 22401 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.318467 22435 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:08.363016 22435 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.362303 22371 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.366003 22560 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.366890 22574 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.356285 22575 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.372512 22575 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.315311 22458 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.351256 22570 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.320441 22410 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.354315 22550 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.355584 22461 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:08.356215 22604 log.cc:927] Time spent T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.287s user 0.000s sys 0.006s
I20251214 02:59:08.359088 22020 raft_consensus.cc:2393] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:08.322448 22440 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.314214 22540 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.320065 22518 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.361941 22556 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.355619 22586 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.359005 22500 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.395051 22547 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.386443 22540 raft_consensus.cc:697] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.377785 22587 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.393925 22550 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.397663 22540 consensus_queue.cc:237] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.398809 22547 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.399860 22587 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.402930 22519 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.404928 22519 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.406451 22586 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:08.407191 22458 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.414063 22435 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.416141 22435 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.273566 21029 rpcz_store.cc:276] 1214 02:59:04.395011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:04.395120 (+ 109us) service_pool.cc:225] Handling call
1214 02:59:08.246096 (+3850976us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:06.825954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:06.874749 (+ 48795us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:06.874789 (+ 40us) write_op.cc:435] Acquired schema lock
1214 02:59:06.874805 (+ 16us) tablet.cc:660] Decoding operations
1214 02:59:06.875218 (+ 413us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:06.875283 (+ 65us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:06.875311 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
1214 02:59:06.875631 (+ 320us) tablet.cc:699] Row locks acquired
1214 02:59:06.875650 (+ 19us) write_op.cc:260] PREPARE: finished
1214 02:59:06.875801 (+ 151us) write_op.cc:270] Start()
1214 02:59:06.875914 (+ 113us) write_op.cc:276] Timestamp: P: 1765681146875754 usec, L: 0
1214 02:59:06.875935 (+ 21us) op_driver.cc:348] REPLICATION: starting
1214 02:59:06.883199 (+ 7264us) log.cc:844] Serialized 918 byte log entry
1214 02:59:07.723450 (+840251us) op_driver.cc:464] REPLICATION: finished
1214 02:59:07.854448 (+130998us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f67129d6e40
1214 02:59:07.962338 (+107890us) write_op.cc:301] APPLY: starting
1214 02:59:07.974943 (+ 12605us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:07.981737 (+ 6794us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:07.981961 (+ 224us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:08.057858 (+ 75897us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:08.058018 (+ 160us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 02:59:08.058738 (+ 720us) write_op.cc:312] APPLY: finished
1214 02:59:08.058977 (+ 239us) log.cc:844] Serialized 28 byte log entry
1214 02:59:08.059685 (+ 708us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:08.059774 (+ 89us) write_op.cc:454] Released schema lock
1214 02:59:08.059819 (+ 45us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:08.060168 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:08.060194 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":222559,"apply.run_cpu_time_us":2901,"apply.run_wall_time_us":110241,"num_ops":2,"prepare.queue_time_us":272691,"prepare.run_cpu_time_us":16723,"prepare.run_wall_time_us":139386,"raft.queue_time_us":433231,"raft.run_cpu_time_us":34744,"raft.run_wall_time_us":185807,"replication_time_us":847386,"spinlock_wait_cycles":10059136,"thread_start_us":728920,"threads_started":4,"wal-append.queue_time_us":176602}]]}
W20251214 02:59:08.448624 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 127493us
W20251214 02:59:08.450055 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 144534us
I20251214 02:59:08.454010 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.462286 21247 leader_election.cc:400] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
W20251214 02:59:08.465250 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 189676us
I20251214 02:59:08.466506 21274 raft_consensus.cc:3060] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.467180 21204 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
W20251214 02:59:08.358433 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 102870us
I20251214 02:59:08.471387 21282 raft_consensus.cc:2368] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:08.474285 21247 leader_election.cc:403] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.490849 22435 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.493096 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.501334 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50516 (request call id 1193) took 1286 ms. Trace:
I20251214 02:59:08.501914 21278 rpcz_store.cc:276] 1214 02:59:07.214435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:07.214624 (+ 189us) service_pool.cc:225] Handling call
1214 02:59:08.500887 (+1286263us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540012edd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:08.501109 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2470289152}
I20251214 02:59:08.512991 22556 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.513374 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.514554 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:08.531006 22586 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.530014 21030 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:08.531350 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.520129 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet bf5cb5c0f4854e4a846c25eb5ce80a9a with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47052
I20251214 02:59:08.534518 22586 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.536844 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.538228 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.540311 22481 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.540180 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.542032 21129 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:08.526787 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 65130.9us
I20251214 02:59:08.527015 22550 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.531956 22556 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:08.543576 22556 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:08.544046 22556 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.522146 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:08.547338 22540 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.547973 22540 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.549319 22540 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.557906 22586 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.532485 22019 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:08.489006 21204 raft_consensus.cc:606] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.560999 21204 raft_consensus.cc:614] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:08.564287 21274 raft_consensus.cc:2468] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:08.572389 21203 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:08.578212 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.543767 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:08.584043 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47399.2us
I20251214 02:59:08.590796 22547 raft_consensus.cc:2764] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251214 02:59:08.592592 22565 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20251214 02:59:08.585777 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.601251 22586 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:59:08.602219 21204 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.602928 22565 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.603868 22565 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:08.604327 22565 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:08.605624 22565 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.607404 21030 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:08.608984 21030 catalog_manager.cc:5654] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:08.570253 22556 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:08.622640 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 44561.5us
W20251214 02:59:08.626987 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 173289us
W20251214 02:59:08.632104 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 88567.8us
I20251214 02:59:08.628372 22582 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.634117 21281 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:08.634853 21207 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:08.636814 22565 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.645040 21204 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:08.629391 21030 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } }
I20251214 02:59:08.651472 21030 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 2 to 3, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:08.653316 21030 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:08.656966 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:08.657827 21132 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.665506 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.666473 22550 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:08.666697 22401 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.666968 22550 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:08.668057 22550 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.669549 22550 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:08.685748 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:08.688691 22621 tablet_replica.cc:333] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:59:08.697613 22556 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.698971 21098 leader_election.cc:400] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
W20251214 02:59:08.700863 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 72286.4us
I20251214 02:59:08.702009 21248 leader_election.cc:400] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.704910 21132 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:08.700387 21099 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.698848 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.707406 22621 raft_consensus.cc:2243] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:08.707531 21207 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.708402 22621 raft_consensus.cc:2272] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:08.708663 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.709522 21206 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:08.710887 22429 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.711459 22429 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.712409 22429 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.712815 21129 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:08.713943 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:08.715248 22556 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.716096 22556 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:08.716233 22621 ts_tablet_manager.cc:1916] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20251214 02:59:08.711032 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 66928.5us
I20251214 02:59:08.718410 22556 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.719388 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.720273 22391 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.721081 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.721892 22401 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.728839 21207 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:08.730161 21098 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:08.730852 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:08.731635 22618 log.cc:927] Time spent T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20251214 02:59:08.732751 22481 consensus_queue.cc:1059] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:08.733876 22481 raft_consensus.cc:3055] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:08.734263 22481 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.736420 22391 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.737015 21986 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:08.737391 22481 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.738052 22560 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.738816 21985 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:08.746974 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.747807 21207 raft_consensus.cc:2368] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.749637 22560 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:08.750897 22560 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:08.740032 22481 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:08.759200 21126 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:08.760525 21127 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:08.763689 22540 raft_consensus.cc:2804] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.764245 22540 raft_consensus.cc:697] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.765190 22540 consensus_queue.cc:237] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:08.766369 22622 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20251214 02:59:08.758980 22556 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:08.758086 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63406.6us
W20251214 02:59:08.780351 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 50421.3us
W20251214 02:59:08.781916 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 51265.1us
I20251214 02:59:08.778138 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:08.795585 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:08.796414 22401 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.796939 22401 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:08.797863 22401 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:08.805135 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 55842.4us
I20251214 02:59:08.809680 22481 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:08.810364 22481 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:08.811530 22481 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.818915 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.822191 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.823858 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.830981 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.829906 22621 ts_tablet_manager.cc:1929] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:59:08.836637 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.845347 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.846153 21203 raft_consensus.cc:2368] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.847838 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.848598 22550 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.862526 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.861603 21247 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.865106 21204 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:08.859853 22621 log.cc:1199] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/bf5cb5c0f4854e4a846c25eb5ce80a9a
I20251214 02:59:08.867448 21206 raft_consensus.cc:2368] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
W20251214 02:59:08.858301 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 119762us
I20251214 02:59:08.865329 22591 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:08.872654 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:08.875535 21200 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:08.900295 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.900543 21204 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251214 02:59:08.901558 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 86124.9us
I20251214 02:59:08.904291 22478 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:08.874825 22565 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.898991 22591 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:08.910778 22586 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.910471 22591 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.917565 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.921458 22429 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:08.925907 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:08.927958 21204 raft_consensus.cc:1275] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251214 02:59:08.927798 22576 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.065s user 0.001s sys 0.001s
I20251214 02:59:08.929198 22613 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.930501 22565 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:08.929111 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.941555 21200 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:08.940388 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.941370 22382 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:59:08.942269 21281 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:08.942517 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.942889 22440 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:08.944322 21282 raft_consensus.cc:2410] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251214 02:59:08.945746 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:08.946235 21282 raft_consensus.cc:2376] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5. Re-sending same reply.
I20251214 02:59:08.946606 21248 leader_election.cc:400] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.946918 21248 leader_election.cc:403] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251214 02:59:08.922003 22429 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:08.948989 22429 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:08.949726 21203 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:08.949939 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:08.952817 21274 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:08.918004 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:08.971797 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 62293.1us
W20251214 02:59:08.974467 22632 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20251214 02:59:08.977964 21281 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:08.981083 21247 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.984062 22550 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.985209 22630 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:08.979187 22586 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:08.994407 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.997947 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:08.998759 21274 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:08.999152 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:09.000213 22588 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:08.995709 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.002955 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet bf5cb5c0f4854e4a846c25eb5ce80a9a (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:09.004853 21248 leader_election.cc:400] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.005159 21248 leader_election.cc:403] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251214 02:59:09.006198 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:08.971566 22429 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20251214 02:59:09.007334 22582 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:09.007553 22553 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:09.010128 22582 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:09.010809 22565 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:09.011158 22582 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.012720 22582 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:09.015338 22588 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:08.998808 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 89258.3us
I20251214 02:59:09.016750 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.020944 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.026890 22565 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:09.028822 22565 raft_consensus.cc:3055] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:09.029179 22565 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:09.030023 22565 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.030825 22553 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:09.032835 21173 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.033475 22613 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:09.035131 21248 leader_election.cc:400] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:09.035441 21248 leader_election.cc:403] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251214 02:59:09.035966 22553 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:09.022855 22630 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:09.023500 22550 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.053828 22553 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:09.054379 22014 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:09.055395 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.053058 21281 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:09.057193 22520 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:09.061928 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.064648 22520 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:09.066862 22427 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20251214 02:59:09.089790 21281 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:09.095113 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.095918 22565 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:09.096164 22520 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:09.097923 21173 leader_election.cc:400] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3.
I20251214 02:59:09.126658 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.130594 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.131261 21126 raft_consensus.cc:2368] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.141597 21281 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:09.144032 22630 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:09.147656 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.148274 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.148840 21132 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:09.149832 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.150601 21126 raft_consensus.cc:2368] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:09.148433 21127 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:09.159238 21132 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:09.161022 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.162524 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.166850 21282 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:09.168313 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:09.168577 21276 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.169145 21125 raft_consensus.cc:2368] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.169764 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:09.170689 21277 raft_consensus.cc:2393] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:09.171957 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.172699 21129 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:09.173771 21278 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.176498 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.177269 21278 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:09.178531 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.179363 22565 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:09.180294 21247 leader_election.cc:400] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.181578 21247 leader_election.cc:400] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:09.181872 21247 leader_election.cc:403] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251214 02:59:09.177237 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 209) took 2341 ms. Trace:
I20251214 02:59:09.182386 22565 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:09.179819 21031 catalog_manager.cc:5654] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:09.185199 21278 raft_consensus.cc:1275] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.186403 21274 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.191200 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:09.187062 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.194516 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:09.187932 21277 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.188386 21129 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:09.197110 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.197638 21124 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:09.200798 22550 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:09.202664 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.189661 21031 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.211650 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.183346 21030 rpcz_store.cc:276] 1214 02:59:06.835999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:06.836159 (+ 160us) service_pool.cc:225] Handling call
1214 02:59:09.177151 (+2340992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:08.780891 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:08.851206 (+ 70315us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:08.851262 (+ 56us) write_op.cc:435] Acquired schema lock
1214 02:59:08.851303 (+ 41us) tablet.cc:660] Decoding operations
1214 02:59:08.852209 (+ 906us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:08.852296 (+ 87us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:08.852337 (+ 41us) tablet.cc:683] Acquiring locks for 6 operations
1214 02:59:08.852941 (+ 604us) tablet.cc:699] Row locks acquired
1214 02:59:08.852975 (+ 34us) write_op.cc:260] PREPARE: finished
1214 02:59:08.853153 (+ 178us) write_op.cc:270] Start()
1214 02:59:08.853300 (+ 147us) write_op.cc:276] Timestamp: P: 1765681148853095 usec, L: 0
1214 02:59:08.853335 (+ 35us) op_driver.cc:348] REPLICATION: starting
1214 02:59:08.854016 (+ 681us) log.cc:844] Serialized 2535 byte log entry
1214 02:59:09.070488 (+216472us) op_driver.cc:464] REPLICATION: finished
1214 02:59:09.128484 (+ 57996us) write_op.cc:301] APPLY: starting
1214 02:59:09.128577 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:09.128741 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:09.128863 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:09.133998 (+ 5135us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:09.134262 (+ 264us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
1214 02:59:09.134321 (+ 59us) write_op.cc:312] APPLY: finished
1214 02:59:09.134618 (+ 297us) log.cc:844] Serialized 52 byte log entry
1214 02:59:09.143621 (+ 9003us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:09.143738 (+ 117us) write_op.cc:454] Released schema lock
1214 02:59:09.143811 (+ 73us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:09.151966 (+ 8155us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:09.152005 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20534,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":56689,"apply.run_cpu_time_us":2883,"apply.run_wall_time_us":23443,"num_ops":6,"prepare.queue_time_us":90438,"prepare.run_cpu_time_us":5234,"prepare.run_wall_time_us":76094,"replication_time_us":205489,"thread_start_us":72700,"threads_started":2}]]}
I20251214 02:59:09.218509 21274 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:09.211941 21031 catalog_manager.cc:5654] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.219282 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.220181 22630 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:09.220247 21031 catalog_manager.cc:5654] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.228268 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:09.229372 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.230073 22550 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:09.230531 22550 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:09.231472 22550 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.233718 22553 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:09.239774 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.240474 21278 raft_consensus.cc:2368] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4.
I20251214 02:59:09.242542 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.250032 21129 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:09.251046 21028 catalog_manager.cc:5654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.257092 21028 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.258618 21028 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.261601 21124 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:09.289880 21125 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:09.295195 21206 raft_consensus.cc:1275] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.297132 22373 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:09.265410 21277 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:09.267181 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.267031 21028 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.269058 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.303841 21028 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:09.304664 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:09.268443 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:09.270961 21204 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.305748 21028 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.275120 21276 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:09.276117 21280 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:09.307144 21028 catalog_manager.cc:5654] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.276721 21130 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:09.289880 21279 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:09.308491 21028 catalog_manager.cc:5654] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.290994 21131 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.309930 21028 catalog_manager.cc:5654] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.320953 21132 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:09.339391 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.340859 22582 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:09.346313 22427 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:09.338284 22373 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251214 02:59:09.351946 22582 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:09.334700 22630 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20251214 02:59:09.350483 22019 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:59:09.354933 22582 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:09.356823 21247 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:09.362648 21247 leader_election.cc:400] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:09.362270 22540 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:59:09.365391 22500 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20251214 02:59:09.361797 22550 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:09.379045 21279 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.380805 21200 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.393903 21132 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:09.394500 21132 raft_consensus.cc:606] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:09.401026 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 44458.4us
I20251214 02:59:09.401530 21200 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:09.402849 22591 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20251214 02:59:09.406478 22429 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:09.409154 22630 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:09.450680 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.453676 21206 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:09.453806 21274 raft_consensus.cc:1275] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.454175 21206 raft_consensus.cc:606] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:09.445866 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 95815.7us
W20251214 02:59:09.513633 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 53593.6us
I20251214 02:59:09.518527 21173 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.520879 21206 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:09.526156 21206 raft_consensus.cc:606] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:09.532686 21030 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 3 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:09.543612 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.509578 22538 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20251214 02:59:09.546291 21205 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.534741 21030 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:09.519418 22429 consensus_queue.cc:1048] T 96ca056dd7594a23a70918fc54c5eb2a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251214 02:59:09.584350 22656 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
W20251214 02:59:09.584941 22658 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20251214 02:59:09.587059 22553 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:09.587672 22553 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.589598 22553 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:09.593286 22520 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:09.593349 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:09.607518 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:09.617028 21207 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:09.608208 21200 raft_consensus.cc:606] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:09.620299 21031 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:09.641248 21031 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:09.648874 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 55286.9us
I20251214 02:59:09.671656 21206 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.631750 22373 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:09.616989 22401 consensus_queue.cc:1059] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20251214 02:59:09.683656 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:09.658202 22591 raft_consensus.cc:2764] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20251214 02:59:09.676698 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251214 02:59:09.711737 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:09.709636 21129 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:09.717720 22401 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:09.688148 22373 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.721668 22540 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:09.729135 22673 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:09.742524 22673 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:09.743424 22673 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.721736 22656 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:09.746649 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:09.741312 22391 raft_consensus.cc:993] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:09.741092 21131 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.740792 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:09.749101 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:09.751098 22656 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.752700 22540 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.754642 22540 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:09.759752 22670 raft_consensus.cc:1081] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
W20251214 02:59:09.741065 22659 log.cc:927] Time spent T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251214 02:59:09.765189 22014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44282
I20251214 02:59:09.765810 22014 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:09.766165 22014 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:09.768757 21205 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:09.769100 21205 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:09.769788 21205 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.770862 21205 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:09.770761 22019 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:09.781276 22373 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:09.762383 22670 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:09.797765 22675 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20251214 02:59:09.799292 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 68688.2us
I20251214 02:59:09.800741 22662 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:09.803460 22654 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:09.809019 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:09.816080 21203 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:09.818516 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:09.823740 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:09.826020 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 351718us
I20251214 02:59:09.827880 21278 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:09.831785 22518 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:09.832334 22518 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.834220 22518 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:09.835616 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:09.836345 21203 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:59:09.838831 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:09.842978 22373 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:09.857460 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:09.858268 21279 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:09.875221 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:09.876276 21125 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:09.881253 22520 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:09.882230 22520 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.891510 22518 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:09.892131 22518 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:09.894482 22518 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:09.894186 21029 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:09.906625 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:09.907217 21124 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:09.895596 22014 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.899654 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:09.903322 21131 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:09.905581 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:09.927764 21030 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 1 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:09.931488 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:09.932075 21200 raft_consensus.cc:3055] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:09.932379 21200 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:09.933133 21200 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.930114 22014 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:09.934006 21200 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:09.940259 21125 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:09.945056 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:09.945812 21279 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:09.963166 22017 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:09.963806 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet e18f39bed60f44f5b8c25ede5d8bba8f with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50512
I20251214 02:59:09.966408 21200 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:09.968009 21986 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:09.978011 22373 raft_consensus.cc:993] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:09.978494 22373 raft_consensus.cc:1081] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:09.974761 22653 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:09.979692 22653 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:09.971987 22672 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251214 02:59:09.980588 22653 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:09.974128 22478 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251214 02:59:09.973186 22401 raft_consensus.cc:697] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:09.994231 22401 consensus_queue.cc:237] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:09.995175 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:09.999028 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 287405us
I20251214 02:59:09.999123 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:10.000339 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:10.001047 22683 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:10.001574 22591 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.002904 22591 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:10.003373 22591 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:10.005012 21277 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:10.005390 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:10.005532 22591 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.009119 21277 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:10.009598 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:10.000368 22518 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:10.009826 21125 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:10.010689 22565 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:10.011055 22565 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:10.011448 22565 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:10.012719 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:10.013295 21125 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251214 02:59:10.013504 22683 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:10.015597 22683 raft_consensus.cc:993] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:10.016036 22673 raft_consensus.cc:1081] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:10.016296 21173 leader_election.cc:400] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4.
I20251214 02:59:10.016603 21173 leader_election.cc:403] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251214 02:59:10.018429 22565 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.022550 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:10.027822 22677 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.028074 22565 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:10.028992 22588 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.027575 22683 consensus_queue.cc:1059] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251214 02:59:10.030300 22683 raft_consensus.cc:3055] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:10.030768 22683 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:10.031620 22683 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.032765 22683 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.032557 22553 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.035445 21205 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251214 02:59:10.035897 21247 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:10.036664 22565 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.038180 21274 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:10.042927 21278 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:10.045163 22565 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:10.052068 21125 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:10.052726 21282 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:10.054029 22588 consensus_queue.cc:1059] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.055058 22588 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:10.055384 22588 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:10.035856 21205 pending_rounds.cc:85] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 2
I20251214 02:59:10.056258 22588 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.056818 22656 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:10.056888 21205 pending_rounds.cc:107] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251214 02:59:10.057399 22588 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:10.051477 22518 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:10.055414 21278 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50516
I20251214 02:59:10.062896 22697 tablet_replica.cc:333] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:59:10.062808 21282 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:10.068608 21282 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.074046 22697 raft_consensus.cc:2243] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:10.075731 21282 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.063949 21278 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:10.081676 22679 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:10.090667 22673 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4.
I20251214 02:59:10.094067 21029 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 2 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:10.083143 21099 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:10.057546 21205 raft_consensus.cc:2889] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:10.087436 21279 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:10.087808 21278 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.088507 21282 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:10.118721 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 69065.9us
I20251214 02:59:10.123204 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:10.123878 21204 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.112682 22697 raft_consensus.cc:2272] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:10.126389 22656 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:10.112643 22553 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20251214 02:59:10.128895 22697 ts_tablet_manager.cc:1916] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20251214 02:59:10.109799 22613 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20251214 02:59:10.141544 21278 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.145466 21278 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:10.120769 22672 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.223017 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:10.324285 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet e18f39bed60f44f5b8c25ede5d8bba8f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20251214 02:59:10.413795 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 289089us
W20251214 02:59:10.326467 22576 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20251214 02:59:10.425693 21201 raft_consensus.cc:3055] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
W20251214 02:59:10.426147 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 283270us
I20251214 02:59:10.444538 22697 ts_tablet_manager.cc:1929] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251214 02:59:10.460760 21130 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:10.464639 21124 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:10.142107 22401 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20251214 02:59:10.446412 22672 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:10.467571 21201 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:10.467674 21018 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 326849us
I20251214 02:59:10.468335 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:10.472466 21201 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.477079 21201 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:10.482924 22705 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.305s user 0.005s sys 0.005s
I20251214 02:59:10.477556 22697 log.cc:1199] T e18f39bed60f44f5b8c25ede5d8bba8f P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/e18f39bed60f44f5b8c25ede5d8bba8f
I20251214 02:59:10.486398 22662 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:10.504544 22661 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20251214 02:59:10.528905 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:10.515489 21201 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:10.516641 21204 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:10.619091 21279 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:10.630867 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:10.627635 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:10.633082 21124 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.633419 21130 raft_consensus.cc:3055] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:10.623557 22713 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:10.633752 21130 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 02:59:10.627559 22683 consensus_meta.cc:287] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.594s user 0.006s sys 0.011s
I20251214 02:59:10.639940 21130 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.643469 21130 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:10.677829 21248 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:10.631309 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:10.642715 22701 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
W20251214 02:59:10.675200 22711 log.cc:927] Time spent T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20251214 02:59:10.711923 22700 log.cc:927] Time spent T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
W20251214 02:59:10.712141 22712 log.cc:927] Time spent T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20251214 02:59:10.531888 21131 raft_consensus.cc:2955] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:10.743753 21124 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:10.714941 21129 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:10.648895 22713 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.751480 21247 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:10.755646 22710 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251214 02:59:10.758396 22715 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:10.758905 22715 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.759797 22518 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:10.760277 22518 raft_consensus.cc:697] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:10.761160 22518 consensus_queue.cc:237] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.762430 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:10.763087 21125 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:10.777896 21247 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:10.749056 22565 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:10.780092 22565 raft_consensus.cc:697] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:10.780959 22565 consensus_queue.cc:237] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.786238 22500 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:10.786857 22500 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:10.788517 22715 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:10.789921 22500 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:10.790230 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:10.789692 22715 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:10.791663 22715 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:10.792032 22715 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:10.798907 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:10.799750 21124 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:10.800801 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:10.800211 22478 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:10.809476 22017 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:10.832363 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 276523us
W20251214 02:59:10.839767 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 215145us
I20251214 02:59:10.844204 21130 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:10.756152 22713 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:10.866947 22717 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:10.867535 22717 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:10.867066 22715 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.867877 22717 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:10.870694 22720 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20251214 02:59:10.873015 22715 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:10.875439 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:10.859884 22678 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:10.884644 21201 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:10.885030 21208 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:10.897037 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:10.898303 21130 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:10.907928 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:10.887362 22723 log.cc:927] Time spent T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20251214 02:59:10.920441 21124 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:10.916077 21129 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:10.922130 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:10.883623 22715 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:10.923342 21131 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:10.928145 21277 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:10.932981 21279 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:59:10.936965 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 90239.9us
I20251214 02:59:10.937369 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:10.885553 22678 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:10.937533 21279 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:10.939237 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:10.885718 22726 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20251214 02:59:10.942250 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:10.951968 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet e18f39bed60f44f5b8c25ede5d8bba8f (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:10.932097 22715 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:10.940518 22518 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:10.980288 22715 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:10.981740 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:10.988384 22678 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:10.998611 22518 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:11.005759 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:11.008225 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:11.010968 21279 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:10.998772 21125 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:10.978689 22718 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20251214 02:59:11.015618 21130 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:11.016342 22019 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.016829 22653 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:11.018720 21247 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:11.017306 21126 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20251214 02:59:11.011456 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 71578.6us
I20251214 02:59:11.030999 22717 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.032037 21282 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:11.035394 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:11.033895 21245 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:11.021852 22553 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
W20251214 02:59:11.052112 22729 log.cc:927] Time spent T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.117s user 0.000s sys 0.002s
I20251214 02:59:11.021283 22518 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.085839 21278 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:11.094326 22730 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:11.096194 21098 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:11.101609 21282 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:11.096356 21276 raft_consensus.cc:2955] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:11.100153 22553 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:11.077718 22401 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20251214 02:59:11.080396 22673 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251214 02:59:11.115061 22429 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:11.114583 22683 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251214 02:59:11.129159 22726 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20251214 02:59:11.112237 22714 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.019s
I20251214 02:59:11.133096 22429 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:11.118548 22518 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:11.132201 22591 raft_consensus.cc:3055] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:11.158090 22591 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:11.160657 21098 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:11.124225 22677 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251214 02:59:11.161473 22429 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:11.135187 22520 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251214 02:59:11.190722 22478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47052 (request call id 836) took 1500 ms. Trace:
I20251214 02:59:11.157029 22676 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251214 02:59:11.191201 22478 rpcz_store.cc:276] 1214 02:59:09.690195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:09.691919 (+ 1724us) service_pool.cc:225] Handling call
1214 02:59:09.750064 (+ 58145us) log.cc:844] Serialized 501 byte log entry
1214 02:59:11.190634 (+1440570us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1159,"raft.run_cpu_time_us":1041,"raft.run_wall_time_us":1160,"thread_start_us":127912,"threads_started":1,"wal-append.queue_time_us":126455,"wal-append.run_cpu_time_us":1582,"wal-append.run_wall_time_us":1009966}
I20251214 02:59:11.193745 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:11.196520 21031 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:11.198642 21031 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:11.200052 21031 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:11.202438 21031 catalog_manager.cc:5654] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:11.197449 22717 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:11.205420 21276 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:11.205304 21031 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:11.171078 22391 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251214 02:59:11.158092 22518 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.164364 22553 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.223824 22518 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:11.231381 22556 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:11.236354 22429 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.243703 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:11.254875 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:11.258548 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 103114us
I20251214 02:59:11.260722 21208 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:11.261411 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:11.263836 21207 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:11.264032 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:11.274121 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:11.255878 22677 consensus_queue.cc:1243] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 6e36caefa37d47a5968cffd9e270044f log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251214 02:59:11.268853 22724 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251214 02:59:11.290854 22717 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:11.293651 22429 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:11.291141 22670 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:11.308828 22678 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:11.309362 22678 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.311051 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:11.311270 22678 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:11.289130 22556 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20251214 02:59:11.317512 21203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "19fd2760f37a43e2983b015db150e285"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:11.315039 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:11.318055 21203 raft_consensus.cc:606] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:11.319034 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:11.319783 21277 raft_consensus.cc:2410] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251214 02:59:11.302461 22726 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:11.327524 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "205cf8366aa44b27a349916da03cff46"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:11.328145 21124 raft_consensus.cc:606] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:11.313349 22670 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.337671 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:11.359624 21130 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:11.373572 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:11.374065 21124 raft_consensus.cc:606] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:11.330374 22726 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.394411 21282 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251214 02:59:11.369717 22662 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:11.399940 22731 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:11.400476 22731 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.401067 22726 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:11.408876 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:11.409575 21124 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:11.406747 21199 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:11.410485 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:11.415458 21126 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:11.415999 21126 raft_consensus.cc:606] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:11.421954 22670 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:11.391685 21125 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:11.402395 22731 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:11.385170 22724 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20251214 02:59:11.437327 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:11.427882 22553 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:11.440503 21277 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251214 02:59:11.452137 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:11.452833 21124 raft_consensus.cc:606] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:11.455943 22591 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.456781 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:11.457007 22591 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:11.457523 21125 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251214 02:59:11.432313 22730 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251214 02:59:11.459210 21201 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:11.462698 22679 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:11.463085 22679 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:11.463253 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:11.465682 21207 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251214 02:59:11.466022 21207 pending_rounds.cc:85] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 2
I20251214 02:59:11.466260 21207 pending_rounds.cc:107] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251214 02:59:11.466472 21207 raft_consensus.cc:2889] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:11.470415 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:11.471096 21278 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:11.472462 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:11.474118 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:11.476258 22714 raft_consensus.cc:993] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:11.477375 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:11.483330 22717 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:11.442116 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:11.442705 22553 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:11.488425 22588 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:11.490402 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:11.479912 22714 raft_consensus.cc:1081] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:11.508775 22722 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20251214 02:59:11.489318 22553 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:11.493695 22588 consensus_queue.cc:1243] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 6e36caefa37d47a5968cffd9e270044f log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251214 02:59:11.526660 22721 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20251214 02:59:11.530735 22556 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:11.536973 22613 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:11.537533 22613 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.542083 22726 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:11.545521 21276 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:11.550436 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:11.555281 21279 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "19fd2760f37a43e2983b015db150e285"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50516
I20251214 02:59:11.555877 21279 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:11.556236 21279 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:11.560438 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:11.567857 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 161930us
I20251214 02:59:11.569691 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:11.581508 21248 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:11.583940 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:11.572513 21279 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.591531 21279 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:11.595461 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:11.610447 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:11.611145 22020 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:11.613234 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:11.610800 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:11.610839 21208 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:11.619827 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:11.617013 21208 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:11.581265 22656 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:11.590205 22730 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:11.566951 22613 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:11.560194 22726 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.581064 22683 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:11.620903 22725 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:11.623174 22720 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:11.624300 22656 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:11.664963 21208 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.623410 22553 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.674060 22656 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:11.686653 22725 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:59:11.687954 22761 log.cc:927] Time spent T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251214 02:59:11.685463 21125 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:11.838533 21208 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:11.631448 22730 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:11.849380 22730 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:11.852473 21279 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20251214 02:59:11.846952 22767 log.cc:927] Time spent T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
W20251214 02:59:11.741698 22700 log.cc:927] Time spent T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251214 02:59:11.875053 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:11.888480 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 263386us
I20251214 02:59:11.837899 22725 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:11.908136 22725 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:11.909137 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 167740us
I20251214 02:59:11.912835 22746 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:11.913235 22746 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:11.914006 22746 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:11.915130 22746 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:11.918367 22020 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:11.922212 22725 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:11.926833 21200 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251214 02:59:11.927563 21200 pending_rounds.cc:85] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 1
I20251214 02:59:11.927829 21200 pending_rounds.cc:107] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251214 02:59:11.928114 21200 raft_consensus.cc:2889] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:11.945431 22715 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:11.965735 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:12.026770 22724 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251214 02:59:12.045787 22553 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:12.028076 22726 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:12.051995 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 87934.1us
I20251214 02:59:12.055020 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:12.054401 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:12.059405 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 171094us
I20251214 02:59:12.061268 21245 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:12.061956 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:12.054909 22779 log.cc:927] Time spent T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20251214 02:59:12.097051 22725 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:12.148759 21208 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:12.148556 22730 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:12.137226 22556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41740 (request call id 1) took 1346 ms. Trace:
I20251214 02:59:12.154942 22725 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:12.172413 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:12.178900 21099 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:12.181344 22481 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:12.188809 22679 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.189846 22679 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:12.190294 22518 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:12.190855 22518 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.191309 22661 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:12.192764 22518 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:12.200284 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:12.165968 22730 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.170485 22556 rpcz_store.cc:276] 1214 02:59:10.778877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:10.779042 (+ 165us) service_pool.cc:225] Handling call
1214 02:59:11.015363 (+236321us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b58004e0bd0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670eefa0da 00007f670ef7c177 00007f670ef9938a 00007f670ef9a3ea 00007f670ef9959b 00007f670efa4cbc 00007f670efa4333 00007f670ef9c67a 00007f6713f53ff9 00007f670d4e5db4
1214 02:59:11.020613 (+ 5250us) log.cc:844] Serialized 501 byte log entry
1214 02:59:12.056602 (+1035989us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":512,"raft.run_cpu_time_us":1038,"raft.run_wall_time_us":1037,"spinlock_wait_cycles":490136448,"thread_start_us":137429,"threads_started":1,"wal-append.queue_time_us":141697}
I20251214 02:59:12.243142 22540 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:12.243770 22540 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:12.245635 22540 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:12.257280 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 123497us
I20251214 02:59:12.262449 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:12.262971 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:12.311858 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:12.257131 22733 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:12.312098 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 172173us
W20251214 02:59:12.312753 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 49321.1us
W20251214 02:59:12.353343 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 48098us
I20251214 02:59:12.259843 21073 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 75c047f1d6d94237afc050db2241d659 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20251214 02:59:12.366933 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45469.9us
I20251214 02:59:12.369539 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.250268 22725 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:12.369504 21073 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 110835us
W20251214 02:59:12.374979 22788 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20251214 02:59:12.354112 22733 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.377101 22733 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:12.378593 22730 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:12.367096 21202 raft_consensus.cc:2955] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:12.379298 22656 consensus_meta.cc:287] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.528s user 0.001s sys 0.007s
I20251214 02:59:12.379681 22656 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.381325 22789 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:12.381577 22656 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:12.381840 22789 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.385619 21132 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:12.390244 21200 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:12.391467 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:12.391119 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:12.393185 21282 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:12.393988 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:12.394552 21132 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:12.392617 22673 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:12.421033 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:12.432638 21099 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:12.434078 22672 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:12.437737 22746 consensus_meta.cc:287] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.522s user 0.005s sys 0.004s
I20251214 02:59:12.439697 22789 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:12.440778 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:12.441107 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.443019 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:12.446354 21132 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:12.454501 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.460618 22762 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:12.416280 22673 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.487363 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.492178 21203 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
W20251214 02:59:12.494550 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 362898us
I20251214 02:59:12.507928 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:12.508823 21130 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:12.513219 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.531911 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:12.545500 22518 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:12.548151 22518 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:12.549978 22518 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:12.550659 22673 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:12.553728 22792 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:12.549923 22747 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:12.554225 22747 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:12.554977 22747 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.555922 22747 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:12.564236 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:12.565029 21279 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:12.545159 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 48250us
I20251214 02:59:12.566501 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:12.568861 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:12.571281 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:12.580756 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:12.581648 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 50768.2us
I20251214 02:59:12.581774 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.583824 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:12.568928 22518 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:12.585498 21247 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:12.582719 22713 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:12.574993 22725 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251214 02:59:12.593438 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:12.587049 22726 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:12.592240 22588 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:12.597538 22518 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
W20251214 02:59:12.730922 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 177258us
I20251214 02:59:12.618005 22809 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:12.776769 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:12.778633 21279 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251214 02:59:12.733484 22518 raft_consensus.cc:697] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:12.782505 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 193356us
W20251214 02:59:12.779325 22807 log.cc:927] Time spent T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251214 02:59:12.826587 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:12.745599 22783 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:12.835265 22478 raft_consensus.cc:993] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:12.835696 22812 raft_consensus.cc:993] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:12.836364 22653 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:12.775519 22804 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20251214 02:59:12.868860 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:12.904022 22653 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:12.904390 22518 consensus_queue.cc:237] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:12.906061 22653 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:12.848533 22783 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:12.908365 22783 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:12.909545 22664 raft_consensus.cc:1081] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:12.912086 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:12.847366 22676 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:12.874922 22478 raft_consensus.cc:1081] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:12.920195 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.007967 21200 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:13.029224 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.035459 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.040630 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.035951 22730 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20251214 02:59:13.045643 22725 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:13.047083 22725 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.047475 22725 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.049746 22656 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:13.050233 22656 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.052100 22656 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.052238 22014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:41740 (request call id 7) took 1514 ms. Trace:
I20251214 02:59:13.052721 22014 rpcz_store.cc:276] 1214 02:59:11.537788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:11.537952 (+ 164us) service_pool.cc:225] Handling call
1214 02:59:13.051676 (+1513724us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b54006bba54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6713f5758e 00007f670d4f6437 00007f670d4f63c2 00007f670d4f62f1 00007f670d4f626c 00007f670d4f5522 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:13.052148 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1655867648}
I20251214 02:59:13.064288 22722 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:13.064797 22722 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.067198 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.070317 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.116550 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.117409 21200 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.103605 22811 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.102131 22722 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.125039 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.127627 22751 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.127669 22747 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.105232 22591 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.106371 22730 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.164901 22747 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:13.128508 22734 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.165982 22734 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.166838 22747 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.181260 21248 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.113070 22731 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.184078 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.184156 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.205544 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.169019 22714 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 02:59:13.220369 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 380395us
I20251214 02:59:13.221604 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:13.225721 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 391554us
I20251214 02:59:13.237445 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:13.169113 22591 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.239861 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.164077 22811 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.182642 22730 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.241286 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.168430 22751 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.199174 22731 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.248462 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.248536 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.237520 22734 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.236982 22714 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.239806 22672 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.254122 22017 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:13.251194 22679 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.242497 22746 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:13.258749 21282 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:13.261298 21282 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:13.261652 21282 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.260046 21279 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "205cf8366aa44b27a349916da03cff46"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:13.263684 21279 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:13.264020 21279 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:13.277217 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.009s sys 0.018s
W20251214 02:59:13.277554 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.009s sys 0.019s
I20251214 02:59:13.255719 22816 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:13.284538 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.268659 22746 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.286623 22746 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.286430 21279 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.282004 22751 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.283099 22811 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.283346 22591 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.283771 22817 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:13.276801 22672 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:13.284205 22730 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.304204 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.271001 22713 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20251214 02:59:13.288843 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.304823 21278 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:13.289078 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.280882 22731 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.284873 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.291426 22725 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.306505 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.294450 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.295089 22740 raft_consensus.cc:993] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:13.295109 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.297829 21282 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.301371 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.301432 21279 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.284483 22736 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:13.310247 21276 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:13.310825 21282 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.312733 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.314440 21202 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.316319 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.319589 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.319610 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.322891 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.319720 21280 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:13.323565 21132 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:13.313730 22736 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:13.325058 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.325273 21204 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.327466 22791 raft_consensus.cc:1081] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:13.316002 22751 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:13.328109 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.336150 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.337643 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.340684 21202 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:13.341732 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.341904 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.346871 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.346909 22679 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:13.347769 22679 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:13.348105 22679 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:13.348968 22801 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.823s user 0.007s sys 0.017s
I20251214 02:59:13.335069 22760 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.337757 22751 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.350988 22751 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.348779 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.347438 21128 raft_consensus.cc:3055] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:13.347060 22811 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:13.354215 21128 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.335214 22736 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.358726 22811 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:13.355587 21125 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:13.360353 21125 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.310330 22672 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:13.363664 22725 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.365362 22672 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:13.359622 22811 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.368402 21128 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.369590 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.373075 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:13.373413 21204 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:13.374461 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.376960 22725 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:13.377434 22725 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:13.378268 22725 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.381214 22751 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.383656 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.384918 22751 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.387413 22736 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.388543 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:13.389101 21275 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.389314 22736 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.393692 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.419220 21202 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.420583 21248 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.389024 22746 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.435101 21204 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251214 02:59:13.479975 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 150389us
I20251214 02:59:13.444445 22736 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:13.633119 21073 catalog_manager.cc:5177] ChangeConfig:REMOVE_PEER RPC for tablet 75c047f1d6d94237afc050db2241d659 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 6
I20251214 02:59:13.565245 21247 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.543087 22820 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.542479 22725 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.636349 22725 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.636143 22820 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.629536 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 104) took 2650 ms. Trace:
W20251214 02:59:13.481956 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 46645us
I20251214 02:59:13.637838 21028 rpcz_store.cc:276] 1214 02:59:10.979164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:10.979321 (+ 157us) service_pool.cc:225] Handling call
1214 02:59:13.629468 (+2650147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1382658,"reported_tablets":1}
I20251214 02:59:13.638199 22725 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.565236 21280 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:13.639303 21280 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:13.569893 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.616431 21031 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 206) took 2805 ms. Trace:
I20251214 02:59:13.640763 21280 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.632766 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 214) took 3497 ms. Trace:
I20251214 02:59:13.546875 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.642860 21030 rpcz_store.cc:276] 1214 02:59:10.134979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:10.143440 (+ 8461us) service_pool.cc:225] Handling call
1214 02:59:10.940529 (+797089us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5400220790. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb3002 00007f6714dbc129 00007f6714d54106 00007f6714d24ff8 00007f6714e1f44e 00007f671331d4a4 00007f671331d432 00007f671331d361 00007f671331d2dc
1214 02:59:13.632697 (+2692168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1387571,"reported_tablets":5,"spinlock_wait_cycles":12332416}
W20251214 02:59:13.618193 22827 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20251214 02:59:13.634953 22736 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.646318 22736 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.519142 22722 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.132s
I20251214 02:59:13.546654 21199 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:13.651328 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.480602 22730 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.654644 22730 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:13.587862 22538 raft_consensus.cc:993] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:13.655987 22538 raft_consensus.cc:1081] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:13.656859 22730 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.654721 21280 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:13.657559 22822 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.337s user 0.008s sys 0.000s
W20251214 02:59:13.657980 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 92822.6us
I20251214 02:59:13.658773 21128 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:13.659312 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 60483.7us
I20251214 02:59:13.659947 21280 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.661725 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.525445 21275 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:13.667052 21125 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.667939 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.667606 21276 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:13.668182 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.668666 21126 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 02:59:13.642891 22797 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.101s user 0.020s sys 0.022s
I20251214 02:59:13.670619 21126 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:13.670639 22818 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:13.672677 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.676735 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.677345 21203 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.683755 22820 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.658730 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:13.684271 21203 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.640347 21031 rpcz_store.cc:276] 1214 02:59:10.811284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:10.811431 (+ 147us) service_pool.cc:225] Handling call
1214 02:59:13.616359 (+2804928us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:11.305051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:11.305569 (+ 518us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:11.305602 (+ 33us) write_op.cc:435] Acquired schema lock
1214 02:59:11.305619 (+ 17us) tablet.cc:660] Decoding operations
1214 02:59:11.306219 (+ 600us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:11.306296 (+ 77us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:11.306322 (+ 26us) tablet.cc:683] Acquiring locks for 5 operations
1214 02:59:11.383760 (+ 77438us) tablet.cc:699] Row locks acquired
1214 02:59:11.384955 (+ 1195us) write_op.cc:260] PREPARE: finished
1214 02:59:11.401775 (+ 16820us) write_op.cc:270] Start()
1214 02:59:11.401905 (+ 130us) write_op.cc:276] Timestamp: P: 1765681151401721 usec, L: 0
1214 02:59:11.401928 (+ 23us) op_driver.cc:348] REPLICATION: starting
1214 02:59:11.402640 (+ 712us) log.cc:844] Serialized 2094 byte log entry
1214 02:59:11.837182 (+434542us) op_driver.cc:464] REPLICATION: finished
1214 02:59:12.037617 (+200435us) write_op.cc:301] APPLY: starting
1214 02:59:12.037708 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:12.037891 (+ 183us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:12.038001 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:12.040201 (+ 2200us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:12.040362 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1214 02:59:12.040404 (+ 42us) write_op.cc:312] APPLY: finished
1214 02:59:12.040619 (+ 215us) log.cc:844] Serialized 47 byte log entry
1214 02:59:12.040776 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:12.040888 (+ 112us) write_op.cc:454] Released schema lock
1214 02:59:12.040928 (+ 40us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:12.041265 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:12.041293 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":200129,"apply.run_cpu_time_us":2141,"apply.run_wall_time_us":3633,"num_ops":5,"prepare.queue_time_us":69724,"prepare.run_cpu_time_us":3540,"prepare.run_wall_time_us":97882,"replication_time_us":290212,"thread_start_us":112065,"threads_started":2}]]}
I20251214 02:59:13.684952 21247 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.686592 22808 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:13.686738 22722 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.660677 21099 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:13.688892 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.692130 22538 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:13.698436 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:13.525974 22831 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251214 02:59:13.668123 21125 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.437611 22518 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:13.707659 22518 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:13.698894 22722 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:13.710055 22722 raft_consensus.cc:697] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:13.712363 22789 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.525104 22746 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.712935 22789 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:13.713385 21031 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.701925 22714 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:13.479830 22731 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.722877 22714 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:13.723073 22731 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.536662 22679 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.723891 22714 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.725703 22679 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.491382 22553 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:13.726689 22553 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:13.708516 21275 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:13.714290 22746 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.716153 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:13.716576 22789 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:13.717087 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.717430 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.717829 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.717722 22722 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.718964 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.708534 22518 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.702028 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 196) took 2600 ms. Trace:
I20251214 02:59:13.741866 21029 rpcz_store.cc:276] 1214 02:59:11.101395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:11.101699 (+ 304us) service_pool.cc:225] Handling call
1214 02:59:13.701971 (+2600272us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":14}
I20251214 02:59:13.743970 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.686266 21247 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.745952 22730 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.735675 21031 catalog_manager.cc:5654] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.747386 21031 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 4 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:13.736239 21199 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:13.750763 21125 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:13.754081 21173 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:13.762136 21128 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:13.762534 22714 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.763446 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.763931 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.764180 22714 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:13.755749 22731 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.757156 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:13.759616 22553 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:13.760483 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:13.765973 21276 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:13.766808 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:13.745941 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 59755.4us
I20251214 02:59:13.772701 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.772945 21276 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:13.774405 22731 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.777184 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.776633 21202 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:13.779911 22820 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:13.782044 22679 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:13.783227 22679 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:13.784338 22679 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.786329 22731 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:13.749516 21031 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:13.801411 21031 catalog_manager.cc:5654] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.808468 22780 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:13.809618 22780 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:13.809983 22780 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.810791 22780 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.812120 22780 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.813902 21274 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:13.815033 22814 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:13.816694 22738 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:13.816255 21031 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.817059 22738 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.817817 22738 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.819630 22738 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:13.824410 22814 consensus_queue.cc:1059] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:13.825183 22814 consensus_queue.cc:1243] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer 6e36caefa37d47a5968cffd9e270044f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251214 02:59:13.826164 22780 raft_consensus.cc:3055] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:13.826704 22780 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.827924 22780 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.829449 21274 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:13.832410 21274 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:13.832736 21274 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.832001 22780 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.840186 21274 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:13.841910 21274 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:13.842887 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:13.843420 21202 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.848989 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.850420 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.853216 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.857173 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.857367 21202 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.858465 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:13.859233 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.859536 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.860461 22722 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:13.860970 22722 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:13.861836 22722 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:13.859565 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:13.873179 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:13.873343 21128 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:13.873713 21130 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:13.874006 21130 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:13.874521 21130 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:13.875469 21130 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:13.878945 21247 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.880483 22811 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.881517 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.882709 22811 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.882929 21133 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.883944 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.885298 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:13.886322 21247 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.887501 21247 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:13.888688 22808 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.889647 22808 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:13.890841 22808 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.881517 21130 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:13.891636 22731 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:13.892817 22731 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:13.939489 21028 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:13.943915 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet 75c047f1d6d94237afc050db2241d659 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47052
I20251214 02:59:13.978737 21031 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.980418 21031 catalog_manager.cc:5654] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:13.984403 21130 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251214 02:59:13.982070 21031 catalog_manager.cc:5654] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:13.985136 22738 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:13.985636 21031 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:13.986088 22738 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:13.987103 22808 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:13.988145 21200 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251214 02:59:13.989001 22738 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:13.989849 22808 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:13.990912 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:13.991266 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:13.991891 21274 raft_consensus.cc:2368] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:13.994966 21098 leader_election.cc:400] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:13.995270 21098 leader_election.cc:403] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:13.998880 22738 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 5
I20251214 02:59:14.005667 22738 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:14.035745 21030 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:14.037668 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b5a42ee941246bf8d0050ea0dcfb9fc with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20251214 02:59:14.046742 22852 tablet_replica.cc:333] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:59:14.047267 21030 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:14.052802 22852 raft_consensus.cc:2243] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:14.054468 22852 raft_consensus.cc:2272] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:14.057577 22852 ts_tablet_manager.cc:1916] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:14.059088 22855 tablet_replica.cc:333] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:59:14.060159 22855 raft_consensus.cc:2243] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:14.061020 22855 raft_consensus.cc:2272] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:14.064196 22855 ts_tablet_manager.cc:1916] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:14.120548 22852 ts_tablet_manager.cc:1929] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20251214 02:59:14.124286 21130 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:14.127552 22852 log.cc:1199] T 75c047f1d6d94237afc050db2241d659 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/75c047f1d6d94237afc050db2241d659
I20251214 02:59:14.128456 22760 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.139117 21279 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:14.140564 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet 75c047f1d6d94237afc050db2241d659 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:14.140887 22746 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.151310 22855 ts_tablet_manager.cc:1929] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20251214 02:59:14.151952 22737 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:14.152424 22737 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.154311 22737 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.154937 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:14.155531 21203 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:14.155612 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:14.156317 21276 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:14.156582 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:14.157419 22737 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:14.158037 22737 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:14.158746 22737 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:14.164011 21028 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:14.166306 21028 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:14.167475 22855 log.cc:1199] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/6b5a42ee941246bf8d0050ea0dcfb9fc
I20251214 02:59:14.170233 22737 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.171725 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 6b5a42ee941246bf8d0050ea0dcfb9fc (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:14.180492 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:14.188277 22737 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.190894 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:14.191661 21279 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:14.187062 21203 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:14.193780 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:14.194658 22737 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:14.195109 22737 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:14.199096 22737 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.212127 21028 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:14.223384 21079 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 668ad4f30bd649c88a034bbab9f13b20 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
W20251214 02:59:14.227895 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9695e153d2f54b008f8c7201463cfbc8, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:14.228369 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9695e153d2f54b008f8c7201463cfbc8, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.231324 22737 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:14.231846 22737 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:14.233080 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 320811ca9d184da38a1197ebfdb7cd0f, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:14.233407 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 320811ca9d184da38a1197ebfdb7cd0f, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.233683 22737 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.235174 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:14.235961 21203 raft_consensus.cc:2410] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251214 02:59:14.237596 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:14.238466 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96ca056dd7594a23a70918fc54c5eb2a, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:14.238922 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96ca056dd7594a23a70918fc54c5eb2a, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.239048 21098 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:14.240206 22737 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:14.245992 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d55cde2542d544c5b99e6113b95910b8, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:14.246313 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d55cde2542d544c5b99e6113b95910b8, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.248695 21203 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.250640 22730 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.257290 21130 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.260509 22731 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.275208 21133 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.278081 22808 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:14.287107 21279 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:14.288411 22760 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:14.288995 22760 consensus_queue.cc:1243] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 6e36caefa37d47a5968cffd9e270044f log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251214 02:59:14.291035 21279 pending_rounds.cc:85] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 1
I20251214 02:59:14.291297 21279 pending_rounds.cc:107] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251214 02:59:14.291538 21279 raft_consensus.cc:2889] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:14.330349 21200 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.332226 22818 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.364871 21200 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.365319 21130 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:14.366858 22714 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.367161 22818 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.404075 21128 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:14.405526 22731 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:14.591243 21276 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:14.592510 22738 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:14.611055 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:14.616662 21200 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:14.619307 22780 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:14.748160 21276 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:14.748716 21276 raft_consensus.cc:606] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:14.751384 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:14.751909 21128 raft_consensus.cc:606] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:14.754279 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:14.754786 21128 raft_consensus.cc:606] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:14.756911 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:14.757376 21128 raft_consensus.cc:606] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:14.763689 21276 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:14.764201 21276 raft_consensus.cc:606] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:14.765313 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:14.766899 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:14.767601 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:14.791105 22738 raft_consensus.cc:993] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:14.791654 22814 raft_consensus.cc:1081] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:14.793095 21200 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:14.793608 21200 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:14.793906 21200 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:14.805522 22760 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:14.806047 22760 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.807891 22760 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.809935 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:14.810717 21279 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:14.809935 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:14.812002 21128 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:14.813438 22814 raft_consensus.cc:993] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:14.814687 22814 raft_consensus.cc:1081] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:14.815061 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:14.816043 22714 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:14.816414 22714 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:14.816761 22714 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.838941 21276 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:14.839596 21276 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:14.839936 21276 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.849943 21276 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.839797 21200 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.856554 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:14.857095 21130 raft_consensus.cc:3055] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:14.857355 21130 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:14.857993 21130 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.859251 21130 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:14.865958 21130 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:14.873544 21173 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:14.839063 22714 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.870023 22800 raft_consensus.cc:993] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:14.874272 22746 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:14.871549 21200 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.874758 22746 raft_consensus.cc:697] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:14.872486 21276 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:14.875646 22746 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.874758 22800 raft_consensus.cc:1081] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:14.877238 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:14.877815 21203 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.878228 22012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44272
I20251214 02:59:14.878803 22012 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:14.879201 22012 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.886032 22012 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:14.888500 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:14.889052 21128 raft_consensus.cc:3055] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:14.889313 21128 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:14.889905 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:14.890460 21133 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.891585 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:14.892127 21279 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:14.897629 21203 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:14.898763 22012 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.899504 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:14.899729 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:14.900032 21277 raft_consensus.cc:3055] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:14.900427 21277 raft_consensus.cc:740] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:14.901126 21277 consensus_queue.cc:260] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:14.902343 21277 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.905018 22714 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:14.906702 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:14.914328 21277 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:14.889945 21128 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.920220 21128 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.936103 21124 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:14.937299 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:14.938385 21133 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:14.938310 22789 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:14.941470 22789 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:14.942312 22789 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.943626 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:14.944345 21276 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:14.945097 21985 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:14.945946 22782 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:14.945122 22808 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:14.947124 22808 raft_consensus.cc:697] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:14.947974 22808 consensus_queue.cc:237] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:14.965370 22782 raft_consensus.cc:697] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:14.972784 22782 consensus_queue.cc:237] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:14.972157 21279 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:14.974911 22730 raft_consensus.cc:993] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:14.976251 22731 raft_consensus.cc:1081] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:14.978344 21203 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:14.979043 21203 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:14.979379 21203 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:14.980525 21128 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:14.987293 21028 catalog_manager.cc:5654] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 3 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:14.996160 21030 catalog_manager.cc:5654] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:14.997823 21031 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:15.020694 21203 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:15.023298 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:15.023936 21128 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:15.024860 21203 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:15.025027 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:15.025710 21276 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:15.025965 21276 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:15.026541 21276 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:15.027688 21276 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:15.033641 21276 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:15.033648 21128 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:15.034658 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:15.035971 22789 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:15.036492 22789 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:15.037377 22789 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:15.068158 21031 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:15.069320 21031 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:15.145396 22778 raft_consensus.cc:993] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:15.145859 22778 raft_consensus.cc:1081] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:15.149493 21279 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:15.150036 21279 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:15.150404 21279 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:15.156497 21279 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:15.159852 21279 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:15.160692 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:15.161255 21203 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:15.164160 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:15.164716 21128 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:15.164983 21128 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:15.165657 21128 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:15.166837 21128 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:15.171919 21203 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:15.172827 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:15.173142 21128 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:15.173743 22730 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:15.174248 22730 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:15.175117 22730 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:15.188048 21031 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:15.328353 21279 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:15.331920 22789 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:59:15.337595 21079 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20251214 02:59:15.343140 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2ce7181915c14c369bac4b962d34913a, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.343611 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2ce7181915c14c369bac4b962d34913a, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.348066 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5bdfefe7cfb64f5cba716a427e83fc9f, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.348439 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5bdfefe7cfb64f5cba716a427e83fc9f, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.353042 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ae31f8b580840eb863e4d400367c1be, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.353423 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ae31f8b580840eb863e4d400367c1be, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.360369 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96ca056dd7594a23a70918fc54c5eb2a, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:15.360745 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96ca056dd7594a23a70918fc54c5eb2a, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:15.373143 21128 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:15.374420 22731 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:15.407501 21128 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:15.412899 22714 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:15.471434 21277 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:15.473142 22789 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.482540 21200 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:15.486152 22730 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:15.560570 21133 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:15.563086 22890 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.570232 21276 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:15.581461 22890 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.585064 21130 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:15.586405 22760 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.610162 21126 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:15.611517 21276 raft_consensus.cc:1275] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:15.633795 21133 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:15.656530 22871 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:15.662668 22893 consensus_queue.cc:1048] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:15.724701 22730 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.728581 21200 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:15.729763 22730 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:15.974071 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:16.035890 21277 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:16.036396 21277 raft_consensus.cc:606] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:16.038539 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:16.039048 21200 raft_consensus.cc:606] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:16.041158 21277 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "19fd2760f37a43e2983b015db150e285"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:16.041554 21277 raft_consensus.cc:606] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:16.043602 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:16.044114 21124 raft_consensus.cc:606] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:16.046265 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:16.046706 21200 raft_consensus.cc:606] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:16.054931 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:16.056234 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:16.056787 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:16.384701 22888 raft_consensus.cc:993] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.385442 22714 raft_consensus.cc:1081] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.387048 22012 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44282
I20251214 02:59:16.387593 22012 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:16.387929 22012 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:16.398407 22888 raft_consensus.cc:993] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.399401 22888 raft_consensus.cc:1081] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.409026 22012 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.412269 22907 raft_consensus.cc:993] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:16.412844 22907 raft_consensus.cc:1081] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:16.414490 21277 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:16.417344 21277 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:16.418027 21277 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:16.427182 22020 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44282
I20251214 02:59:16.427760 22020 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:16.428074 22020 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:16.427410 22012 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:16.427207 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:16.431228 21128 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:16.433290 21277 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.434821 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:16.435343 21200 raft_consensus.cc:3055] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:16.435614 21200 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:16.436255 21277 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:16.436270 21200 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.438033 21200 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:16.438668 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:16.439169 21203 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:16.443078 21079 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251214 02:59:16.445092 21200 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:16.446215 21986 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:16.447165 22891 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:16.447137 22020 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:16.448016 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 275e8242241b4fd386a7c5c738a2c903, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:16.448518 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 275e8242241b4fd386a7c5c738a2c903, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:16.454396 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2ca27c31805c4e55898fe4f60468ffa6, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:16.454762 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2ca27c31805c4e55898fe4f60468ffa6, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.456234 21128 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:16.458040 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:16.458689 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 205cf8366aa44b27a349916da03cff46, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:16.462669 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 205cf8366aa44b27a349916da03cff46, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.462602 21124 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:16.463793 21124 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:16.460011 22891 raft_consensus.cc:697] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:16.462025 22800 raft_consensus.cc:993] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:16.464449 21124 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.465512 21124 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:16.466892 21203 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:16.464443 22800 raft_consensus.cc:1081] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:16.468006 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:16.468786 22731 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:16.469231 22731 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:16.469077 22891 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.470045 22731 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.471488 21199 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:16.471997 21199 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:16.472304 21199 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:16.474632 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:16.475180 21133 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:16.479197 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f692dd0fa4b64450b9808a14da2b1c29, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:16.479499 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f692dd0fa4b64450b9808a14da2b1c29, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.481962 21199 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.485823 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:16.486900 21124 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:16.485913 21133 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:16.486467 21128 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:16.488039 22020 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:16.488555 21985 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:16.490217 22910 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:16.488472 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:16.491374 21203 raft_consensus.cc:3055] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:16.492496 21203 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:16.493345 21203 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.494091 21128 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:16.486923 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:16.495101 21277 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:16.495436 21277 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:16.496098 21277 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.497164 21277 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:16.498632 21199 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:16.500842 22910 raft_consensus.cc:697] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:16.502697 22910 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.503530 21028 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:16.495213 21203 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:16.512826 21277 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:16.513859 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:16.515602 22714 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:16.516059 22714 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:16.516834 22714 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:16.525295 21029 catalog_manager.cc:5654] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 3 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:16.526875 21203 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:16.543090 21031 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:16.578995 22800 raft_consensus.cc:993] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.579743 22731 raft_consensus.cc:1081] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:16.581786 22020 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "19fd2760f37a43e2983b015db150e285"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44272
I20251214 02:59:16.582307 22020 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:16.582774 22020 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:16.593323 22020 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.597867 22020 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:16.598636 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:16.599198 21274 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:16.599462 21274 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:16.599822 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:16.600353 21200 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:16.600099 21274 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.601336 21274 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:16.608363 21274 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:16.608695 21200 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:16.609920 21985 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:16.610738 22910 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:16.610023 21029 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 2 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:16.611227 22910 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:16.612075 22910 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:16.628271 21029 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 5 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:16.892542 21200 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:16.897105 22910 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:16.915558 21128 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:16.915778 21200 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:16.917642 22920 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:16.918587 22891 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:16.923240 21274 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:16.924786 22888 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:16.948240 21200 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:16.949929 22891 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:16.955444 21133 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:16.957768 22714 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:16.988662 21133 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:16.990262 22731 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.007006 21128 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:17.008608 22912 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:17.057054 21274 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:17.058176 22891 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:17.068738 21203 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:17.070241 22910 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:17.093026 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:17.136770 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:17.137434 21200 raft_consensus.cc:606] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:17.139693 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "143de581216646f79bece2b9f1259f64"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:17.140228 21128 raft_consensus.cc:606] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:17.142547 21274 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:17.143533 21274 raft_consensus.cc:606] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:17.145764 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:17.146165 21128 raft_consensus.cc:606] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:17.148005 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:17.148381 21128 raft_consensus.cc:606] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:17.149165 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:17.150241 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:17.150775 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:17.251524 22937 raft_consensus.cc:993] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:17.252012 22908 raft_consensus.cc:1081] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:17.253422 21200 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "143de581216646f79bece2b9f1259f64"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:17.253926 21200 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:17.254231 21200 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:17.259516 21200 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.261576 21200 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:17.262037 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:17.262640 21128 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:17.262897 21128 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:17.263585 21128 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.264379 22738 raft_consensus.cc:993] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.264635 21128 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:17.264845 22937 raft_consensus.cc:1081] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.266302 21277 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:17.266978 21277 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:17.267648 21277 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:17.268712 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:17.269137 21274 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:17.274395 21128 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:17.277742 21274 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:17.278705 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:17.279804 22888 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:17.281035 22888 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:17.282162 22888 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.286228 21277 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.290398 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:17.290951 21128 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:17.291208 21128 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:17.291911 21128 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.292503 21277 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:17.293054 21128 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:17.293521 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:17.294026 21200 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:17.298115 21200 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:17.299149 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:17.299437 21128 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:17.299980 22800 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:17.300530 22800 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:17.301383 22800 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.310856 21029 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:17.314591 21028 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:17.381680 22800 raft_consensus.cc:993] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:17.382170 22731 raft_consensus.cc:1081] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:17.383445 21200 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:17.383956 21200 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:17.384284 21200 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.390748 21200 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.392140 21200 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:17.392720 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:17.393270 21128 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.393455 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:17.393947 21274 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:17.394214 21274 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:17.394876 21274 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.395830 21274 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.400496 21128 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:17.402626 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:17.403404 22909 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:17.403837 22909 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:17.404587 22909 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.408330 21274 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:17.426606 21028 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:17.462474 22888 raft_consensus.cc:993] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.462927 22909 raft_consensus.cc:1081] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.464130 21277 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50516
I20251214 02:59:17.464643 21277 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:17.464955 21277 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.471590 21277 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.473449 21277 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:17.489229 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:17.489882 21133 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.494478 22908 raft_consensus.cc:993] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.495003 22908 raft_consensus.cc:1081] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:17.495141 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:17.496277 21200 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:17.510552 21200 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:17.506778 21133 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:17.496731 21274 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:17.511798 21274 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:17.513005 21274 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:17.513589 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:17.516287 21200 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.517495 21200 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:17.519739 21274 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:17.521765 21274 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:17.524226 22946 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:17.524706 22946 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:17.525472 22946 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:17.549080 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:17.549702 21128 raft_consensus.cc:3055] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:17.549999 21128 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:17.551083 21128 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:17.552090 21128 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:17.559803 21128 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:17.559455 21028 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:17.560779 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:17.561537 22946 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:17.561995 22946 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:17.562834 22946 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:17.570365 21200 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:17.591657 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:17.597805 22020 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:17.599702 21029 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:17.604776 22020 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:17.643891 21133 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:17.656041 21200 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20251214 02:59:17.656297 21096 consensus_peers.cc:597] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fdf7633f868447a8abd2adfe9b04fc5. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:17.657282 21274 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:17.657289 22908 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:17.659024 22738 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.668051 22738 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.669574 21274 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.683629 21207 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.691246 21028 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.698395 21133 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:17.706357 21277 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:59:17.707413 21096 consensus_peers.cc:597] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b1dc02b9fd94ebd9df45c7602bc4cbd. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:17.708461 22738 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.710709 21207 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:17.712091 22738 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.726683 22937 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.728240 21277 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.743530 21276 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:17.740322 21207 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.747285 22888 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:17.743391 21031 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:17.742398 21274 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:17.774242 21133 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:17.777173 22714 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251214 02:59:17.825418 21245 consensus_peers.cc:597] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 320811ca9d184da38a1197ebfdb7cd0f. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:17.828330 21133 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:17.831475 22731 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:17.828330 21200 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:17.850376 22947 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:17.857368 22945 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.863052 21200 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:17.864336 22731 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.861405 21128 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.876506 21199 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:17.890164 21124 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20251214 02:59:17.891041 21171 consensus_peers.cc:597] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 143de581216646f79bece2b9f1259f64. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:17.891546 22909 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.897615 22714 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.898659 21279 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:17.905046 21028 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:17.936965 21124 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:17.939528 22945 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:17.951330 21130 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:17.962576 21207 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.974742 22714 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:17.982805 22020 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:17.984110 21130 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:17.982504 22888 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:17.988615 22945 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:18.005014 21199 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:18.006393 21277 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:18.010190 21128 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:18.012549 22947 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251214 02:59:18.014914 21277 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:18.021651 21028 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:18.041981 22947 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20251214 02:59:18.046252 22971 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20251214 02:59:18.104058 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 61619.9us
I20251214 02:59:18.107066 21125 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:18.110589 22731 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251214 02:59:18.133762 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 72286.3us
I20251214 02:59:18.189234 22981 ts_tablet_manager.cc:933] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:59:18.191938 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:18.207304 22982 ts_tablet_manager.cc:933] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:59:18.227582 22981 tablet_copy_client.cc:323] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:59:18.229779 21143 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 9fdf7633f868447a8abd2adfe9b04fc5 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:59:18.230360 21143 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 9fdf7633f868447a8abd2adfe9b04fc5 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-9fdf7633f868447a8abd2adfe9b04fc5
I20251214 02:59:18.237326 22982 tablet_copy_client.cc:323] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:59:18.239171 21142 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:59:18.239689 21142 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-1b1dc02b9fd94ebd9df45c7602bc4cbd
I20251214 02:59:18.241760 21143 tablet_copy_source_session.cc:215] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:18.244998 22981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fdf7633f868447a8abd2adfe9b04fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:18.257347 21142 tablet_copy_source_session.cc:215] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:18.260968 22982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:18.285560 22982 tablet_copy_client.cc:806] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:18.286204 22982 tablet_copy_client.cc:670] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:18.286222 22981 tablet_copy_client.cc:806] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:18.286978 22981 tablet_copy_client.cc:670] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:18.287387 21125 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:18.288022 21125 raft_consensus.cc:606] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:18.293066 21277 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:18.293572 21277 raft_consensus.cc:606] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:18.295219 22982 tablet_copy_client.cc:538] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:18.296180 21207 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:18.296703 21207 raft_consensus.cc:606] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:18.304930 21125 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:18.305495 21125 raft_consensus.cc:606] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:18.295987 22981 tablet_copy_client.cc:538] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:18.306648 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 02:59:18.309986 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:18.310642 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:18.313827 22908 raft_consensus.cc:993] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:18.314288 22908 raft_consensus.cc:1081] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:18.316751 21277 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:18.317250 21277 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:18.317552 21277 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:18.318886 22982 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:18.324462 22985 ts_tablet_manager.cc:933] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:18.335999 22981 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:18.339123 21277 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.341944 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:18.342499 21125 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:18.342823 21125 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:18.342901 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:18.343643 21203 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:18.343611 21125 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.347537 21125 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:18.351315 22985 tablet_copy_client.cc:323] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:59:18.355357 21277 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:18.352969 21292 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet 320811ca9d184da38a1197ebfdb7cd0f from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:59:18.358173 21125 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:18.358691 21292 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet 320811ca9d184da38a1197ebfdb7cd0f from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-320811ca9d184da38a1197ebfdb7cd0f
I20251214 02:59:18.356603 21203 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:18.360190 21248 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:18.360924 22948 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:18.361379 22948 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:18.363190 22948 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.374747 21292 tablet_copy_source_session.cc:215] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:18.378299 22985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320811ca9d184da38a1197ebfdb7cd0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:18.385092 21028 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:18.404155 22985 tablet_copy_client.cc:806] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:18.404686 22985 tablet_copy_client.cc:670] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:18.408368 22985 tablet_copy_client.cc:538] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:18.417026 22985 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:18.433768 22982 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:18.434662 22982 tablet_bootstrap.cc:492] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:18.435345 22982 ts_tablet_manager.cc:1403] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.117s user 0.031s sys 0.014s
I20251214 02:59:18.438097 22982 raft_consensus.cc:359] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.438917 22982 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:18.439589 22982 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.447651 22990 ts_tablet_manager.cc:933] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:18.450484 22985 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:18.452034 22985 tablet_bootstrap.cc:492] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:18.452677 22985 ts_tablet_manager.cc:1403] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.036s user 0.031s sys 0.001s
I20251214 02:59:18.462349 22985 raft_consensus.cc:359] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.463151 22985 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:18.463742 22985 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.466125 22985 ts_tablet_manager.cc:1434] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.013s user 0.001s sys 0.003s
I20251214 02:59:18.468665 21292 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-320811ca9d184da38a1197ebfdb7cd0f received from {username='slave'} at 127.0.0.1:50140
I20251214 02:59:18.469074 21292 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-320811ca9d184da38a1197ebfdb7cd0f on tablet 320811ca9d184da38a1197ebfdb7cd0f with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:18.478811 22990 tablet_copy_client.cc:323] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:59:18.481014 22982 ts_tablet_manager.cc:1434] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.045s user 0.003s sys 0.008s
I20251214 02:59:18.484030 22981 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:18.510834 22981 tablet_bootstrap.cc:492] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:18.511646 22981 ts_tablet_manager.cc:1403] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.176s user 0.027s sys 0.005s
I20251214 02:59:18.518232 22981 raft_consensus.cc:359] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.519062 22981 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:18.519250 21142 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-1b1dc02b9fd94ebd9df45c7602bc4cbd received from {username='slave'} at 127.0.0.1:52098
I20251214 02:59:18.519997 21142 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-1b1dc02b9fd94ebd9df45c7602bc4cbd on tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:18.519693 22981 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:18.522313 22981 ts_tablet_manager.cc:1434] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20251214 02:59:18.523747 21142 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-9fdf7633f868447a8abd2adfe9b04fc5 received from {username='slave'} at 127.0.0.1:52098
I20251214 02:59:18.524144 21142 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-9fdf7633f868447a8abd2adfe9b04fc5 on tablet 9fdf7633f868447a8abd2adfe9b04fc5 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:18.527673 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 143de581216646f79bece2b9f1259f64 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:59:18.528211 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 143de581216646f79bece2b9f1259f64 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-143de581216646f79bece2b9f1259f64
I20251214 02:59:18.532372 22955 raft_consensus.cc:993] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:18.532826 22955 raft_consensus.cc:1081] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:18.537216 22949 raft_consensus.cc:993] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:18.537673 22949 raft_consensus.cc:1081] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:18.545468 21203 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:18.546023 21203 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:18.544811 21207 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:18.546325 21203 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:18.546712 21207 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:18.547070 21207 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:18.547755 21218 tablet_copy_source_session.cc:215] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:18.551715 22990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143de581216646f79bece2b9f1259f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:18.553267 21203 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.553758 21207 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.555142 21203 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:18.555445 21207 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:18.556352 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:18.558224 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:18.561432 21125 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:18.560487 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:18.562992 21277 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:18.563328 21277 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:18.564045 21277 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.565765 21277 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:18.567612 21125 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:18.565142 21124 raft_consensus.cc:3055] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:18.568135 21124 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:18.568902 21124 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.570955 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:18.571671 22714 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:18.572090 22714 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:18.572826 22714 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.575786 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:18.577569 21277 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:18.577965 21274 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:18.583528 22990 tablet_copy_client.cc:806] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:18.584062 22990 tablet_copy_client.cc:670] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:18.585624 21274 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:18.619316 21124 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:18.636721 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:18.639420 21027 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:18.675709 22714 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:18.676585 22714 raft_consensus.cc:697] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:18.677778 22714 consensus_queue.cc:237] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.697571 21124 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
W20251214 02:59:18.777158 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 51452.7us
I20251214 02:59:18.779964 22990 tablet_copy_client.cc:538] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:18.793452 21027 catalog_manager.cc:5654] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:18.797523 22990 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:18.804170 21124 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:18.805586 22800 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:18.813755 22020 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Advancing to term 4
I20251214 02:59:18.814786 22012 raft_consensus.cc:1217] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
W20251214 02:59:18.820163 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56618.8us
I20251214 02:59:18.825275 22012 raft_consensus.cc:1217] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251214 02:59:18.838636 22020 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20251214 02:59:18.840997 23002 log.cc:927] Time spent T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251214 02:59:18.892944 22800 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:18.912978 22995 raft_consensus.cc:993] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:18.913522 22995 raft_consensus.cc:1081] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
W20251214 02:59:18.921149 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 76500.2us
I20251214 02:59:18.932181 21130 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:18.932929 21130 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:18.933347 21130 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:18.956724 21200 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251214 02:59:18.958287 22731 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:18.961442 21130 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:18.990325 21130 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:18.991441 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:18.992076 21274 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:18.993305 22949 raft_consensus.cc:1064] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:18.995925 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:18.996483 21200 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:18.996735 21200 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:18.996910 22949 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:18.997407 21200 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:19.008220 22990 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:19.009155 22990 tablet_bootstrap.cc:492] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:19.009881 22990 ts_tablet_manager.cc:1403] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.214s user 0.050s sys 0.010s
I20251214 02:59:19.012717 21133 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:19.013720 21200 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:19.015239 22997 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:19.025732 21274 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:19.026983 21098 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:19.038718 21279 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:19.043133 21280 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:19.050657 22998 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:19.051230 22998 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:19.052224 22998 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:19.060129 22995 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:19.068415 21200 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:19.079296 22012 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:19.082001 21207 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:19.090354 22947 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251214 02:59:19.091841 22949 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.094647 22990 raft_consensus.cc:359] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:19.095783 22990 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:19.096416 22990 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:19.103336 22990 ts_tablet_manager.cc:1434] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.093s user 0.003s sys 0.007s
I20251214 02:59:19.104918 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-143de581216646f79bece2b9f1259f64 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:59:19.105369 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-143de581216646f79bece2b9f1259f64 on tablet 143de581216646f79bece2b9f1259f64 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:19.108330 22949 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.115778 21202 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.123234 21130 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.141376 21027 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.147670 21133 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:19.148509 22012 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.155815 22998 raft_consensus.cc:1064] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:19.155725 22971 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:19.159040 22998 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.176195 22012 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.179912 22998 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.134346 21028 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:19.184540 21276 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.186209 22937 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:19.191439 22997 raft_consensus.cc:1064] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:19.194084 21203 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.194304 22997 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.198843 22998 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.200248 21274 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:19.212925 22013 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:19.216179 22949 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.232215 23005 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251214 02:59:19.249264 21133 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:19.253937 21199 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:19.256314 22997 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.258980 22947 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.266309 21277 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.271201 21133 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:19.273201 23005 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:19.273686 22998 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.275568 22013 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.282910 21207 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.288585 22020 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:19.292913 21124 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:19.301115 22996 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:19.307314 22020 raft_consensus.cc:1217] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251214 02:59:19.308516 22996 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:19.324493 22996 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.351639 21124 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.354802 22949 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.359179 21028 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.360216 21199 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.369369 21130 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.388407 22013 raft_consensus.cc:2955] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.389604 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:19.394714 21274 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.411007 22012 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.419843 21133 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.428781 21029 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 4 to 5, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:19.427078 21030 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 5 to 6, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.448493 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:19.451093 21274 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.456538 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42016
I20251214 02:59:19.464133 22013 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:19.467058 21207 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:19.468797 22937 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.471349 21199 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:19.473423 22013 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:19.476838 22998 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:19.480738 22996 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:19.493597 22947 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.494527 22738 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.495786 21199 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.504940 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 320811ca9d184da38a1197ebfdb7cd0f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:19.504258 21203 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.508019 23027 tablet_replica.cc:333] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:59:19.510473 23027 raft_consensus.cc:2243] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:19.512353 23027 raft_consensus.cc:2272] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:19.512142 22949 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:19.513587 22013 raft_consensus.cc:2955] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.521387 22012 raft_consensus.cc:2955] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:19.523623 21030 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 5 to 6, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:19.525846 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 9fdf7633f868447a8abd2adfe9b04fc5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:19.530884 23027 ts_tablet_manager.cc:1916] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:19.545877 21028 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:19.568812 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fdf7633f868447a8abd2adfe9b04fc5 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50512
I20251214 02:59:19.580947 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet 320811ca9d184da38a1197ebfdb7cd0f with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47052
I20251214 02:59:19.595764 23029 tablet_replica.cc:333] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:59:19.616333 23030 tablet_replica.cc:333] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:59:19.617218 23030 raft_consensus.cc:2243] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:19.617581 23027 ts_tablet_manager.cc:1929] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20251214 02:59:19.621824 23027 log.cc:1199] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/1b1dc02b9fd94ebd9df45c7602bc4cbd
I20251214 02:59:19.622193 23030 raft_consensus.cc:2272] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:19.624300 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet 1b1dc02b9fd94ebd9df45c7602bc4cbd (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:19.624619 21274 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:19.626435 23029 raft_consensus.cc:2243] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:19.627251 23029 raft_consensus.cc:2272] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:19.627956 22738 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.634188 23029 ts_tablet_manager.cc:1916] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:19.636133 21207 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:19.636724 21207 raft_consensus.cc:606] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:19.640782 23030 ts_tablet_manager.cc:1916] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:19.643579 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:19.644107 21199 raft_consensus.cc:606] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:19.648597 21203 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:19.704681 21133 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:19.722498 22937 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.728331 21133 raft_consensus.cc:606] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:19.728757 21133 raft_consensus.cc:630] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting request to transfer leadership because tablet server 6e36caefa37d47a5968cffd9e270044f is not a voter in the active config
I20251214 02:59:19.753204 21276 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:19.753777 21276 raft_consensus.cc:606] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:19.757910 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 02:59:19.759646 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:19.760569 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:19.768889 23030 ts_tablet_manager.cc:1929] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20251214 02:59:19.769368 23030 log.cc:1199] T 320811ca9d184da38a1197ebfdb7cd0f P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/320811ca9d184da38a1197ebfdb7cd0f
I20251214 02:59:19.775594 23029 ts_tablet_manager.cc:1929] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20251214 02:59:19.779156 23029 log.cc:1199] T 9fdf7633f868447a8abd2adfe9b04fc5 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/9fdf7633f868447a8abd2adfe9b04fc5
I20251214 02:59:19.779369 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet 320811ca9d184da38a1197ebfdb7cd0f (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:19.784751 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 9fdf7633f868447a8abd2adfe9b04fc5 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:19.845995 23004 raft_consensus.cc:993] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:19.846449 23004 raft_consensus.cc:1081] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:19.848227 22013 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44282
I20251214 02:59:19.848762 22013 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:19.849125 22013 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.854601 22995 raft_consensus.cc:1064] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:19.857196 22995 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.869792 22013 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.872957 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:19.872843 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:19.873591 21277 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.873694 21203 raft_consensus.cc:3055] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:19.875891 21203 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:19.876801 22013 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:19.877254 21203 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.880044 21203 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.881511 21277 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:19.882673 21985 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:19.883478 23028 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:19.883927 23028 raft_consensus.cc:697] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:19.884765 23028 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.891279 21203 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:19.929136 22013 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.930552 22969 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.942922 21276 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.955041 21130 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:19.957525 22800 raft_consensus.cc:993] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:19.957976 22800 raft_consensus.cc:1081] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:19.958364 22969 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:19.959405 21202 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:19.959911 21202 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:19.960268 21202 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.959997 21028 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 2, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 2 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:19.966645 21202 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.960160 22969 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20251214 02:59:19.969628 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:19.970021 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:19.970227 21276 raft_consensus.cc:3055] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:19.970535 22012 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.978338 22012 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:19.979557 21171 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:19.980686 21276 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:19.982873 21276 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:19.983944 21276 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:19.981232 23005 raft_consensus.cc:2804] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:19.987362 23005 raft_consensus.cc:697] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:19.988245 23005 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:20.001854 21202 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:20.008148 23004 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.008910 22012 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.029021 21027 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.050133 21276 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:20.049621 21027 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:20.067663 21276 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.070294 21130 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.084772 21199 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:20.091180 22012 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.093715 21130 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.094542 23004 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.102840 23004 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.108510 22012 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.113974 21018 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 143de581216646f79bece2b9f1259f64 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:20.116864 23005 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.121342 21130 raft_consensus.cc:2955] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:20.126866 21027 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:20.141325 23005 raft_consensus.cc:993] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:20.143723 23005 raft_consensus.cc:1081] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:20.145124 21133 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:20.145610 21133 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:20.145924 21133 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:20.149395 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 143de581216646f79bece2b9f1259f64 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50512
I20251214 02:59:20.150398 23029 tablet_replica.cc:333] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:59:20.151381 23029 raft_consensus.cc:2243] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:20.152012 23029 raft_consensus.cc:2272] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:20.157424 23029 ts_tablet_manager.cc:1916] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:20.157311 21133 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:20.159168 21133 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:20.160480 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:20.161108 21207 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:20.161381 21207 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:20.162022 21207 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:20.163230 21207 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:20.160157 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:20.165436 22012 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:20.170529 21207 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:20.171345 22012 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:20.173408 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:20.174223 22937 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:20.174710 22937 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:20.175532 22937 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:20.178901 23029 ts_tablet_manager.cc:1929] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20251214 02:59:20.179369 23029 log.cc:1199] T 143de581216646f79bece2b9f1259f64 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/143de581216646f79bece2b9f1259f64
I20251214 02:59:20.180899 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 143de581216646f79bece2b9f1259f64 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:20.193071 21027 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:20.336784 21202 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.337782 23001 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.358980 21276 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.361310 23028 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.427774 22012 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.428923 23040 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.437644 21279 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:20.438719 23040 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:20.586479 22013 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:20.587781 22738 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:20.601528 21207 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:20.618454 22937 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:20.857900 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:20.906754 21279 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:20.907335 21279 raft_consensus.cc:606] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:20.909626 21207 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:20.910039 21207 raft_consensus.cc:606] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:20.912146 21133 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:20.912664 21133 raft_consensus.cc:606] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:20.914829 21133 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:20.915319 21133 raft_consensus.cc:606] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:20.916268 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 02:59:20.917387 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:20.917922 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:21.106362 23050 raft_consensus.cc:993] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:21.106801 23050 raft_consensus.cc:1081] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:21.108057 21207 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:21.108557 21207 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:21.108888 21207 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.117087 21207 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:21.118702 21207 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:21.119750 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:21.120334 21130 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.120985 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:21.121547 21279 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:21.121801 21279 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:21.122478 21279 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:21.123636 21279 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.125684 21130 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:21.126642 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:21.127252 23005 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:21.127621 23005 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:21.128919 23005 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:21.132575 21279 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:21.140969 21027 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 6 to 7, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 7 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:21.218490 21277 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.224331 21207 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:21.226080 21130 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:21.227023 22731 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251214 02:59:21.228021 21245 consensus_peers.cc:597] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55cde2542d544c5b99e6113b95910b8. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:21.231822 22996 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:21.248739 22731 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.256633 21199 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.257903 21133 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.269209 21277 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.274439 21027 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:21.296337 21245 consensus_peers.cc:597] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3c57218e1ae40dfbdcec8dd228432ca. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:21.298151 21130 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:21.298537 21199 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:21.299880 22731 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:21.302484 22996 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:21.307713 23050 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.316285 21130 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.323648 21199 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.331316 21202 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
W20251214 02:59:21.349118 21171 consensus_peers.cc:597] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd36aadbf4841c9a26e48e96430c009. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:21.351779 21277 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:21.353722 23040 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:21.361769 21027 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 4, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.378196 21133 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251214 02:59:21.379683 23040 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:21.421595 23005 raft_consensus.cc:993] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:21.422108 23005 raft_consensus.cc:1081] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:21.432153 21124 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:21.432683 21124 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:21.433012 21124 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:21.437618 22738 raft_consensus.cc:993] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:21.438118 22738 raft_consensus.cc:1081] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:21.440629 23040 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.443882 21276 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:21.444427 21276 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:21.444756 21276 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:21.444773 21277 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.449049 21126 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.450352 21124 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:21.455013 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:21.455567 21279 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:21.458879 22955 raft_consensus.cc:993] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:21.459323 22955 raft_consensus.cc:1081] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:21.457271 21124 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:21.460856 21274 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:21.461398 21274 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:21.461731 21274 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.464146 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:21.464627 21199 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:21.464900 21199 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:21.465520 21199 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:21.466981 21276 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:21.467741 21274 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:21.469329 21274 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:21.469652 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:21.470146 22013 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:21.475041 21276 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:21.475183 21279 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:21.483732 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:21.484323 21125 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:21.484668 21125 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:21.485311 21125 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:21.486501 21125 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.489738 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:21.490343 21130 raft_consensus.cc:3055] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:21.490635 21130 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:21.491511 21130 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:21.492805 21130 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:21.496896 21027 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:21.500125 22013 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:21.496932 21126 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.501334 21245 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:21.501950 21125 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:21.502544 22996 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:21.506323 22996 raft_consensus.cc:697] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:21.507908 22996 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:21.484366 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:21.512970 22937 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:21.504877 21199 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:21.516937 21130 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:21.518149 22937 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:21.528381 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:21.529104 21202 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:21.528080 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:21.532471 22731 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:21.532917 22731 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:21.533681 22731 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:21.543334 21096 consensus_peers.cc:597] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ae31f8b580840eb863e4d400367c1be. This is attempt 1: this message will repeat every 5th retry.
I20251214 02:59:21.545248 21277 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:21.547046 23005 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:21.548633 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:21.522713 22937 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:21.556046 21199 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:21.560761 21027 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 2 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:21.561928 21277 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:21.565130 22738 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:21.588685 21199 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:21.590103 22738 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20251214 02:59:21.608362 21027 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:21.620282 23070 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.645525 21277 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.672041 21027 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:21.675951 21199 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:21.679517 23078 ts_tablet_manager.cc:933] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:21.681439 21124 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:21.683773 23005 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:21.688349 23078 tablet_copy_client.cc:323] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:59:21.699671 21028 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 3, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:21.711009 21292 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet d55cde2542d544c5b99e6113b95910b8 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:59:21.711596 21292 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet d55cde2542d544c5b99e6113b95910b8 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-d55cde2542d544c5b99e6113b95910b8
I20251214 02:59:21.741796 21292 tablet_copy_source_session.cc:215] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:21.738754 23082 ts_tablet_manager.cc:933] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:21.756932 23078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55cde2542d544c5b99e6113b95910b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:21.767458 23082 tablet_copy_client.cc:323] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.66:44049
I20251214 02:59:21.768935 21218 tablet_copy_service.cc:140] P 668ad4f30bd649c88a034bbab9f13b20: Received BeginTabletCopySession request for tablet 3fd36aadbf4841c9a26e48e96430c009 from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:47944)
I20251214 02:59:21.769497 21218 tablet_copy_service.cc:161] P 668ad4f30bd649c88a034bbab9f13b20: Beginning new tablet copy session on tablet 3fd36aadbf4841c9a26e48e96430c009 from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:47944: session id = 8de0cf3ca4bf4083bc494a9af0e49342-3fd36aadbf4841c9a26e48e96430c009
I20251214 02:59:21.777050 21218 tablet_copy_source_session.cc:215] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:21.780431 23082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd36aadbf4841c9a26e48e96430c009. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:21.781147 23078 tablet_copy_client.cc:806] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:21.781735 23078 tablet_copy_client.cc:670] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:21.785476 23078 tablet_copy_client.cc:538] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:21.795946 23078 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:21.824602 23078 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:21.825636 23078 tablet_bootstrap.cc:492] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:21.826431 23078 ts_tablet_manager.cc:1403] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20251214 02:59:21.827069 23084 ts_tablet_manager.cc:933] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:21.829502 23078 raft_consensus.cc:359] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.830490 23078 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:21.831228 23078 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.837267 23078 ts_tablet_manager.cc:1434] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251214 02:59:21.839599 23084 tablet_copy_client.cc:323] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:59:21.840852 21292 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-d55cde2542d544c5b99e6113b95910b8 received from {username='slave'} at 127.0.0.1:50140
I20251214 02:59:21.841303 21292 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-d55cde2542d544c5b99e6113b95910b8 on tablet d55cde2542d544c5b99e6113b95910b8 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:21.842851 21291 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet a3c57218e1ae40dfbdcec8dd228432ca from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:59:21.843391 21291 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet a3c57218e1ae40dfbdcec8dd228432ca from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-a3c57218e1ae40dfbdcec8dd228432ca
I20251214 02:59:21.846686 23082 tablet_copy_client.cc:806] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:21.847250 23082 tablet_copy_client.cc:670] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:21.851238 23082 tablet_copy_client.cc:538] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:21.858884 21291 tablet_copy_source_session.cc:215] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:21.863754 23084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c57218e1ae40dfbdcec8dd228432ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:21.868464 23082 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:21.880070 23084 tablet_copy_client.cc:806] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:21.880683 23084 tablet_copy_client.cc:670] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:21.884222 23084 tablet_copy_client.cc:538] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:21.894263 23084 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:21.945803 23084 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:21.946832 23084 tablet_bootstrap.cc:492] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:21.947557 23084 ts_tablet_manager.cc:1403] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.029s
I20251214 02:59:21.950521 23084 raft_consensus.cc:359] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.951427 23084 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:21.952112 23084 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.952855 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:21.954849 23084 ts_tablet_manager.cc:1434] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251214 02:59:21.957358 21291 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-a3c57218e1ae40dfbdcec8dd228432ca received from {username='slave'} at 127.0.0.1:50140
I20251214 02:59:21.957789 21291 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-a3c57218e1ae40dfbdcec8dd228432ca on tablet a3c57218e1ae40dfbdcec8dd228432ca with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:21.971822 23082 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:21.978672 23082 tablet_bootstrap.cc:492] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:21.981130 23082 ts_tablet_manager.cc:1403] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.113s user 0.026s sys 0.005s
I20251214 02:59:21.985540 23082 raft_consensus.cc:359] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.987371 23082 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:21.988031 23082 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:21.990659 23082 ts_tablet_manager.cc:1434] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20251214 02:59:21.992468 21218 tablet_copy_service.cc:342] P 668ad4f30bd649c88a034bbab9f13b20: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-3fd36aadbf4841c9a26e48e96430c009 received from {username='slave'} at 127.0.0.1:47944
I20251214 02:59:21.992869 21218 tablet_copy_service.cc:434] P 668ad4f30bd649c88a034bbab9f13b20: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-3fd36aadbf4841c9a26e48e96430c009 on tablet 3fd36aadbf4841c9a26e48e96430c009 with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:22.018502 21124 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:22.023229 22996 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:22.044381 22013 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:22.047295 23063 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:22.056205 21126 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:22.059751 22996 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:22.071291 23082 ts_tablet_manager.cc:933] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855)
I20251214 02:59:22.079747 22012 raft_consensus.cc:1217] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:59:22.097265 23082 tablet_copy_client.cc:323] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.65:38855
I20251214 02:59:22.098955 21142 tablet_copy_service.cc:140] P 721d604130924d939d291fe3eef8f6d1: Received BeginTabletCopySession request for tablet 6ae31f8b580840eb863e4d400367c1be from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:52098)
I20251214 02:59:22.099512 21142 tablet_copy_service.cc:161] P 721d604130924d939d291fe3eef8f6d1: Beginning new tablet copy session on tablet 6ae31f8b580840eb863e4d400367c1be from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:52098: session id = 8de0cf3ca4bf4083bc494a9af0e49342-6ae31f8b580840eb863e4d400367c1be
I20251214 02:59:22.103116 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:22.103781 21199 raft_consensus.cc:606] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:22.123896 21274 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:22.124467 21274 raft_consensus.cc:606] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:22.126752 21199 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:22.130889 23063 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:22.135167 21274 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:22.135731 21274 raft_consensus.cc:606] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:22.139179 21142 tablet_copy_source_session.cc:215] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:22.142374 23082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae31f8b580840eb863e4d400367c1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:22.155071 21126 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:22.155678 21126 raft_consensus.cc:606] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:22.156294 21207 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:59:22.158007 23070 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:22.157050 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 02:59:22.164937 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:22.165680 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:22.172185 21277 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:59:22.173691 22955 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:22.206125 23082 tablet_copy_client.cc:806] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:22.206820 23082 tablet_copy_client.cc:670] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:22.211278 23082 tablet_copy_client.cc:538] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:22.302736 23082 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:22.303853 22731 raft_consensus.cc:993] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:22.305094 23050 raft_consensus.cc:1081] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:22.310957 21207 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:22.311479 21207 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:22.311805 21207 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:22.320272 21207 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:22.343978 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 112801us
I20251214 02:59:22.348059 21207 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:22.362017 22012 raft_consensus.cc:1217] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251214 02:59:22.368356 22013 raft_consensus.cc:1217] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20251214 02:59:22.378283 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:22.379442 21274 raft_consensus.cc:3055] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:22.379851 21274 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:22.380628 21274 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:22.384764 21274 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:22.403604 23050 raft_consensus.cc:993] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:22.434397 23099 raft_consensus.cc:993] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:22.435263 23101 raft_consensus.cc:1081] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:22.437896 21276 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:22.438390 21276 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:22.438752 21276 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.442071 21274 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:22.429178 23103 raft_consensus.cc:1081] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:22.445641 21124 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47078
I20251214 02:59:22.446159 21124 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:22.446476 21124 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.453397 21124 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:22.457083 21124 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:22.475198 21276 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:22.476389 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:22.481629 21277 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:22.481988 21277 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:22.482867 21277 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
W20251214 02:59:22.483824 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 56955.9us
I20251214 02:59:22.484567 21277 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.478731 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:22.485440 21124 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:22.497852 21277 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:22.498493 21276 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:22.498966 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:22.500512 21202 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.506523 21098 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:22.507931 21124 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:22.509478 23050 raft_consensus.cc:1064] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:22.510794 21202 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:22.517521 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:22.518091 21124 raft_consensus.cc:3055] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:22.518354 21124 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:22.519156 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:22.519739 22013 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.520467 21124 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:22.521618 21124 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:22.524084 23107 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:22.524595 23107 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:22.525475 23107 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:22.527052 23050 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:22.528180 23113 raft_consensus.cc:993] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:22.528719 23005 raft_consensus.cc:1081] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:22.528187 21124 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:22.534158 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:22.535104 22731 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:22.535581 22731 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:22.536386 22731 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:22.538823 21173 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:22.539584 23067 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:22.540040 23067 raft_consensus.cc:697] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:22.540846 23067 consensus_queue.cc:237] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:22.556476 22013 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:22.560667 22012 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:22.562646 22996 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:22.570638 21207 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:22.571887 22996 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:22.575341 21124 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:22.576906 22996 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:22.611723 23082 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 02:59:22.612596 23082 tablet_bootstrap.cc:492] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 02:59:22.612038 21028 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:22.613299 23082 ts_tablet_manager.cc:1403] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.311s user 0.022s sys 0.041s
I20251214 02:59:22.616029 23082 raft_consensus.cc:359] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:22.616807 23082 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 02:59:22.617408 23082 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:22.636571 21126 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:22.637212 21126 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:22.637557 21126 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:22.635836 21029 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:22.705142 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45411.7us
W20251214 02:59:22.712653 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 71085.8us
I20251214 02:59:22.717496 21126 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:22.722500 21126 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:22.724452 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 120696us
I20251214 02:59:22.730929 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:22.731467 21274 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:22.746621 21274 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:22.747792 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:22.749055 23082 ts_tablet_manager.cc:1434] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.135s user 0.007s sys 0.001s
I20251214 02:59:22.749148 23099 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:22.749720 23099 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:22.787113 21142 tablet_copy_service.cc:342] P 721d604130924d939d291fe3eef8f6d1: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-6ae31f8b580840eb863e4d400367c1be received from {username='slave'} at 127.0.0.1:52098
I20251214 02:59:22.787632 21142 tablet_copy_service.cc:434] P 721d604130924d939d291fe3eef8f6d1: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-6ae31f8b580840eb863e4d400367c1be on tablet 6ae31f8b580840eb863e4d400367c1be with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:22.750614 23099 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:22.842290 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 122467us
W20251214 02:59:22.852620 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 52422.1us
W20251214 02:59:22.885538 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 89555.7us
I20251214 02:59:22.899569 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:22.900133 21207 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:22.900357 21207 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:22.901044 21126 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:22.908452 21207 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:22.947490 21207 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:22.969385 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45808.9us
I20251214 02:59:22.980211 23126 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:22.969265 21030 catalog_manager.cc:5654] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:23.044176 21207 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:23.044531 21124 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:23.072755 22012 raft_consensus.cc:1217] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251214 02:59:23.074869 22013 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:23.075965 22731 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20251214 02:59:23.083706 22013 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:23.087765 23135 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20251214 02:59:23.107326 22020 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Advancing to term 4
I20251214 02:59:23.109081 21280 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:23.113524 23101 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:23.106431 21124 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:23.115584 21125 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:23.116466 21200 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:23.119616 22731 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:23.125584 22020 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:23.118991 23100 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20251214 02:59:23.119009 23111 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20251214 02:59:23.154753 23114 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:23.204687 23102 raft_consensus.cc:1064] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:23.231999 21279 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:23.317503 21274 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:59:23.320866 21207 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:23.324059 21204 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251214 02:59:23.230326 23102 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:23.345901 21027 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:23.357688 23115 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20251214 02:59:23.369624 23107 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:23.396821 23112 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:23.439114 22012 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:23.441462 23112 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:23.770330 23114 raft_consensus.cc:1064] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:23.773120 23114 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.663465 23162 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.664036 23162 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.684796 23162 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.706766 23162 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.708243 23162 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.709751 23166 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.710817 23162 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.710435 23166 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.713636 23166 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.714928 23166 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.715373 23167 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.715409 23166 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.716003 23167 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.717451 23166 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.717769 23167 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.718624 23169 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.719376 23169 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.721261 23169 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.737891 23163 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.738474 23163 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.740002 23164 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.740490 23163 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.740620 23164 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.742549 23164 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.740104 23162 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:24.746214 23162 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.748214 23162 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:24.756050 23177 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.756587 23177 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.758144 23178 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.758845 23178 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.758802 23177 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.761260 23178 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.764060 23177 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.764547 23177 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.766244 23177 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.768689 23176 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:24.769433 23171 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.769913 23171 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.771894 23171 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.772962 23171 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.773311 21277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1604) took 1362 ms. Trace:
I20251214 02:59:24.769157 23176 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.775089 23176 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:24.773478 23171 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.773711 21277 rpcz_store.cc:276] 1214 02:59:23.410609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:23.411596 (+ 987us) service_pool.cc:225] Handling call
1214 02:59:23.411860 (+ 264us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:23.412153 (+ 293us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:23.412178 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:23.420151 (+ 7973us) log.cc:844] Serialized 26 byte log entry
1214 02:59:23.456455 (+ 36304us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:23.462405 (+ 5950us) log.cc:844] Serialized 14 byte log entry
1214 02:59:23.462640 (+ 235us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:23.462715 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:24.773085 (+1310370us) raft_consensus.cc:1685] finished
1214 02:59:24.773106 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:24.773235 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":884,"prepare.queue_time_us":6557,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":694,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":6379,"thread_start_us":22467,"threads_started":1,"wal-append.queue_time_us":68822,"wal-append.run_cpu_time_us":2553,"wal-append.run_wall_time_us":1004386}
I20251214 02:59:24.781589 23171 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:24.782369 23172 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.782876 23172 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.784260 23178 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:24.784780 23178 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.784701 23172 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.786972 23182 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.787524 23182 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.791177 23169 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.791754 23169 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.793172 23180 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.793602 23169 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.793715 23168 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.793577 23179 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.794263 23168 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.794993 23179 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.796372 23168 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.796989 23179 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.800298 23178 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:24.802078 23176 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.802641 23176 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.804196 23179 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.806546 23186 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.807263 23187 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.807061 23186 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.807735 23187 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.807616 23172 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.809293 23173 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.809136 23172 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:24.809800 23173 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:24.812906 23162 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.813460 23173 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.813416 23162 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.804507 23176 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.811744 23172 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.806219 23184 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:24.816150 23182 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:24.816936 23175 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.816042 23163 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.809775 23187 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.017139 23185 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.017293 23182 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:24.949875 23168 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.793687 23180 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:24.822294 23178 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:24.818428 23179 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:25.018016 23169 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.018462 23181 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:24.809079 23186 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.290000 23171 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.048213 23166 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.051982 23187 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.282622 23167 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.305399 23167 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.307436 23167 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.307947 23167 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.308416 23167 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.310252 23167 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.024660 23175 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.312765 23175 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.292057 23181 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.315410 23181 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.297761 23179 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:24.912680 23184 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.322119 23184 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.299407 23180 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.285130 23182 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.324646 23182 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.300217 23162 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.300593 23187 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.291285 23178 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.327945 23178 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.052278 23163 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.329814 23163 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.288653 23169 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.283119 23168 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.358981 23168 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.302459 23166 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.361707 23166 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.285550 23185 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.363656 23185 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.304469 23165 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.364779 23165 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.366865 23165 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.367347 23187 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.357162 23169 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:25.375442 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.031s sys 0.028s
W20251214 02:59:25.541350 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s user 0.032s sys 0.028s
W20251214 02:59:25.530643 21096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.007s sys 0.013s
W20251214 02:59:25.545222 21096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s user 0.007s sys 0.013s
I20251214 02:59:25.551527 22020 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:25.290841 23171 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.559875 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.561206 23190 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.561709 23190 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.562093 23171 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.547609 23186 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.562738 23186 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.564354 23186 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.564873 23192 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.565373 23192 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:25.549839 23165 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.566664 23165 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.568398 23165 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.569563 23187 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.570030 23178 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.570254 23177 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.570531 23178 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.570748 23177 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.571933 23178 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.572289 23178 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.572559 23178 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.572615 23177 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.573179 23177 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.573582 23177 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.573815 23178 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.553699 23188 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.553544 23173 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.552481 23181 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.575024 23177 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.575070 23173 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.581008 23173 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.576598 23163 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.582635 23163 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.584559 23163 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.578734 23183 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.585579 23183 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.587452 23183 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.587849 23181 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.589761 23181 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.574607 23188 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.591904 23188 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.569741 23185 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.592597 23185 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.595597 23190 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.570044 23187 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.597808 23187 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.599385 23192 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.600172 23186 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:25.631860 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.000s sys 0.017s
W20251214 02:59:25.632200 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.000s sys 0.017s
I20251214 02:59:25.600694 23186 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:25.636137 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.000s sys 0.024s
W20251214 02:59:25.636471 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.000s sys 0.024s
I20251214 02:59:25.638079 23185 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.638549 23186 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.631511 23192 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.639051 23186 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.639277 23192 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.639534 23186 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.641240 23192 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.641937 23177 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.641732 23192 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.642460 23177 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.642735 23192 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.644374 23177 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.641973 23187 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.644968 23187 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:25.645504 23163 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.645970 23163 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.646919 23187 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.647680 23197 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.648201 23197 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.648736 23162 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.649242 23162 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.650274 23197 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.650673 23179 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.651175 23179 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.652868 23192 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.652989 23179 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.614591 23175 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.653605 23175 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.655411 23175 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.655886 23175 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.656328 23175 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.632056 23165 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.657449 23165 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.647778 23163 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.659318 23165 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.641255 23186 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.665243 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.665671 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.666249 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.666534 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.666911 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.666971 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.667649 21207 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:25.673038 23171 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.673554 23171 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.674021 23194 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.674510 23194 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.675491 23171 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.676448 23194 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.682049 23162 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.685529 23165 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.686013 23165 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.687645 23175 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.687788 23165 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.688797 23163 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.688984 23197 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.689308 23163 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.689477 23197 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:25.691186 23197 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.691628 23197 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.691951 23162 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.692058 23197 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.692492 23162 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:25.694057 23197 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.694281 23162 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.702703 23165 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.841403 23165 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.843621 23165 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.703183 23192 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.844290 23192 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.846225 23192 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.703594 23194 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.847426 23194 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.849408 23194 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.713804 23195 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.850066 23195 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.852072 23195 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.714448 23171 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.852720 23171 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.854648 23171 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.719597 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.857004 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.858196 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.858917 22011 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:25.724704 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.726295 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.729199 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:25.812758 19986 debug-util.cc:398] Leaking SignalData structure 0x7b080000f520 after lost signal to thread 21173
W20251214 02:59:25.860666 19986 kernel_stack_watchdog.cc:198] Thread 21173 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 1389ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20251214 02:59:25.691193 23163 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:25.721138 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.862457 22019 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:25.903414 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.904093 21207 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:25.906867 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.929263 23167 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.929874 23167 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.931797 23167 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:25.931954 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.511s user 0.034s sys 0.056s
W20251214 02:59:25.932456 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.511s user 0.034s sys 0.056s
W20251214 02:59:25.932740 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 2.51201e+06us
I20251214 02:59:25.933755 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.933501 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:25.935004 21125 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
W20251214 02:59:25.935130 21098 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:25.935708 23173 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.936187 23173 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.937969 23173 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.938385 23173 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.939005 21277 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:25.939296 23181 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:25.939790 23181 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.939002 23173 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.941641 23181 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.941771 23173 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.950508 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.946964 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:25.945950 22041 debug-util.cc:398] Leaking SignalData structure 0x7b0800339520 after lost signal to thread 19984
I20251214 02:59:25.951342 21277 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:25.942039 23175 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.952106 23175 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.953357 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.953917 21277 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:25.954027 23175 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.942001 23196 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.954869 23196 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.955768 23188 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.956246 23188 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.956719 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.946920 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.957516 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.957641 21201 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:25.958109 23188 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.958667 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.959117 21199 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:25.959200 23188 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:25.959726 23188 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.961134 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.961689 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.962527 23188 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.962980 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.963651 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.965477 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.966084 23190 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.966687 23181 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.966887 23183 raft_consensus.cc:1064] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:25.966607 23190 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.967500 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.968060 21274 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:25.968745 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.968590 23181 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.969327 21279 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:25.969154 23183 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:25.970428 23181 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:25.964184 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.971048 21206 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:25.971318 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.966614 21202 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:25.971931 21279 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:25.972218 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.972764 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.973860 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.976059 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.977084 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.963053 23188 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20251214 02:59:25.978610 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.980253 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.980315 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.980561 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.980630 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.981442 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.982085 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.981185 21279 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 1.
I20251214 02:59:25.982728 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.989369 21280 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:25.983700 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.989728 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.990037 23188 raft_consensus.cc:1027] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: attempt to promote peer 8de0cf3ca4bf4083bc494a9af0e49342: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20251214 02:59:25.990489 21206 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:25.990176 21274 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:25.983672 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.991945 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.983726 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.970378 23190 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:25.984951 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.992652 22020 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:25.985630 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.992702 21208 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:25.986907 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:25.993789 22015 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:25.987004 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.965946 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.995210 21278 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:25.997093 23190 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:25.998399 23190 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:25.999243 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.000355 23190 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.000610 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.001161 21125 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:26.001859 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.002027 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.003108 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:25.987604 23164 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:25.984982 23167 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.004277 23164 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:26.004521 23167 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.014986 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.022485 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.023294 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.023481 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.024251 23167 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.002058 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.006095 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.025450 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.007225 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.007692 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.008085 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.008291 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.008469 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.028406 21276 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:26.028496 21277 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:26.028610 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.010915 23175 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.011847 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.011440 21028 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:26.029443 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.029999 21278 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:26.029839 23175 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.030961 21208 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:26.031733 23175 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.032256 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.032893 21203 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:26.026947 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.034214 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.012769 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.013783 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.034960 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.035027 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.013953 23190 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.036394 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.036558 23190 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.035476 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:26.037894 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.038843 23190 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.013720 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.015056 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.015523 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:25.990456 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.015667 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.016249 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.016923 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.058674 23167 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:26.014442 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 153) took 2792 ms. Trace:
I20251214 02:59:26.059063 23167 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.059365 21201 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:26.060216 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.059012 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.018488 21245 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.059104 23129 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.062366 23129 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.062930 23199 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.063462 23199 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.059255 21027 rpcz_store.cc:276] 1214 02:59:23.222132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:23.222266 (+ 134us) service_pool.cc:225] Handling call
1214 02:59:26.014364 (+2792098us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:23.357408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:23.363664 (+ 6256us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:23.363699 (+ 35us) write_op.cc:435] Acquired schema lock
1214 02:59:23.363714 (+ 15us) tablet.cc:660] Decoding operations
1214 02:59:23.364044 (+ 330us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:23.364096 (+ 52us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:23.364121 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:23.364284 (+ 163us) tablet.cc:699] Row locks acquired
1214 02:59:23.364304 (+ 20us) write_op.cc:260] PREPARE: finished
1214 02:59:23.364455 (+ 151us) write_op.cc:270] Start()
1214 02:59:23.364552 (+ 97us) write_op.cc:276] Timestamp: P: 1765681163364415 usec, L: 0
1214 02:59:23.364571 (+ 19us) op_driver.cc:348] REPLICATION: starting
1214 02:59:23.445459 (+ 80888us) log.cc:844] Serialized 599 byte log entry
1214 02:59:24.283869 (+838410us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b54000cadd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef99aa8 00007f67129d64d1 00007f67129d5ca8 00007f67129da941 00007f67129da8f9 00007f67129da889 00007f67129da851 00007f67129d9a7d 0000000000379254 0000000000379099
1214 02:59:24.733857 (+449988us) op_driver.cc:464] REPLICATION: finished
1214 02:59:24.735515 (+ 1658us) write_op.cc:301] APPLY: starting
1214 02:59:24.735603 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:24.735699 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:24.735718 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:24.735967 (+ 249us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:24.736113 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:24.736160 (+ 47us) write_op.cc:312] APPLY: finished
1214 02:59:24.742502 (+ 6342us) log.cc:844] Serialized 23 byte log entry
1214 02:59:24.749364 (+ 6862us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:24.749461 (+ 97us) write_op.cc:454] Released schema lock
1214 02:59:24.749499 (+ 38us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:24.749863 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:24.749905 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":442265,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":1403,"apply.run_cpu_time_us":7961,"apply.run_wall_time_us":14304,"num_ops":1,"prepare.queue_time_us":3460,"prepare.run_cpu_time_us":16188,"prepare.run_wall_time_us":930177,"replication_time_us":1369122,"spinlock_wait_cycles":99427712,"thread_start_us":9001,"threads_started":2,"wal-append.queue_time_us":6823,"wal-append.run_cpu_time_us":964,"wal-append.run_wall_time_us":1000990}]]}
I20251214 02:59:26.019373 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.020371 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.064308 23129 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.063445 21096 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.024434 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.065333 21096 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.059602 23167 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.066056 23190 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.066443 23190 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.066819 23190 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.063126 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.068641 22018 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEARNER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:26.069873 23188 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.070289 23188 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.031397 23164 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:26.070775 23188 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:25.956707 23196 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:26.070901 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.012105 23109 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.052613 21279 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:26.071276 23112 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251214 02:59:26.071579 22041 debug-util.cc:398] Leaking SignalData structure 0x7b0800339be0 after lost signal to thread 21065
W20251214 02:59:26.064013 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 45593.6us
I20251214 02:59:26.071607 23109 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:26.072824 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002dd740 after lost signal to thread 21157
W20251214 02:59:26.073107 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002ddb20 after lost signal to thread 21158
I20251214 02:59:26.073894 23109 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.073946 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.036649 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.055210 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.020296 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.058038 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.056584 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.057192 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.076201 21245 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.077679 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.078312 21124 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
W20251214 02:59:26.058133 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 67768.2us
I20251214 02:59:26.063899 23199 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.056085 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:26.079269 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.079811 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.061004 21201 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:26.012558 23198 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.034399 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 279) took 3143 ms. Trace:
W20251214 02:59:26.074054 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002f6400 after lost signal to thread 21232
W20251214 02:59:26.085947 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002f62a0 after lost signal to thread 21233
I20251214 02:59:26.086195 21279 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:26.077702 21203 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:26.079493 23192 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:26.083007 23175 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:26.084345 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.087193 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.092094 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.088203 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff380 after lost signal to thread 21306
I20251214 02:59:26.090924 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.090956 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.087230 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.092662 22041 debug-util.cc:398] Leaking SignalData structure 0x7b080018a280 after lost signal to thread 21307
I20251214 02:59:26.091385 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.083755 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.092690 21133 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
W20251214 02:59:26.096741 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002bf000 after lost signal to thread 22044
W20251214 02:59:26.097023 22041 debug-util.cc:398] Leaking SignalData structure 0x7b08002f8540 after lost signal to thread 22045
I20251214 02:59:26.099277 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.100198 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.100953 21173 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.104377 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.104997 21133 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:26.106498 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.125638 23187 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:26.126045 23187 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.126396 23187 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.130137 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.132373 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.106199 23198 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.135195 21171 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.136046 23112 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:26.137116 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:26.091732 23089 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:26.138638 23177 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.138943 23089 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.139134 23177 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:26.139585 23177 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.140018 21125 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 1.
I20251214 02:59:26.142783 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.085114 21030 rpcz_store.cc:276] 1214 02:59:22.891108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:22.891432 (+ 324us) service_pool.cc:225] Handling call
1214 02:59:26.034344 (+3142912us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:23.032538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:23.033117 (+ 579us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:23.033150 (+ 33us) write_op.cc:435] Acquired schema lock
1214 02:59:23.033169 (+ 19us) tablet.cc:660] Decoding operations
1214 02:59:23.033496 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:23.033542 (+ 46us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:23.033575 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:23.033742 (+ 167us) tablet.cc:699] Row locks acquired
1214 02:59:23.033758 (+ 16us) write_op.cc:260] PREPARE: finished
1214 02:59:23.033886 (+ 128us) write_op.cc:270] Start()
1214 02:59:23.033985 (+ 99us) write_op.cc:276] Timestamp: P: 1765681163033841 usec, L: 0
1214 02:59:23.034004 (+ 19us) op_driver.cc:348] REPLICATION: starting
1214 02:59:23.065418 (+ 31414us) log.cc:844] Serialized 541 byte log entry
1214 02:59:23.451873 (+386455us) op_driver.cc:464] REPLICATION: finished
1214 02:59:23.459216 (+ 7343us) write_op.cc:301] APPLY: starting
1214 02:59:23.459288 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:23.459394 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:23.459509 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:23.459724 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:23.459871 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:23.459909 (+ 38us) write_op.cc:312] APPLY: finished
1214 02:59:23.460077 (+ 168us) log.cc:844] Serialized 23 byte log entry
1214 02:59:23.460222 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:23.460298 (+ 76us) write_op.cc:454] Released schema lock
1214 02:59:23.460349 (+ 51us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:23.460649 (+ 300us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:23.460675 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1577,"apply.run_cpu_time_us":1433,"apply.run_wall_time_us":1433,"num_ops":1,"prepare.queue_time_us":111,"prepare.run_cpu_time_us":2542,"prepare.run_wall_time_us":51046,"replication_time_us":413750}]]}
I20251214 02:59:26.087699 23192 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.106460 23129 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:26.148072 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.124002 23185 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.168196 21099 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.166826 21171 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.139433 23089 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.136188 23198 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.147910 23112 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:26.182458 21099 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.184587 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.185081 21128 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:26.186122 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.179881 23183 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.187824 23183 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.188186 23183 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.190441 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.190809 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.191295 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 110173us
I20251214 02:59:26.192221 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.190809 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.167522 23129 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.192699 21126 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:26.168987 23185 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.196101 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.199337 21125 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:26.197680 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.171552 23173 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:26.200603 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 57900.6us
I20251214 02:59:26.200629 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.201782 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.202231 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.203567 21278 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:26.203996 21278 raft_consensus.cc:606] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Received request to transfer leadership to TS 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.191121 23198 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.169353 23192 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.184693 23112 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.207724 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.199168 23129 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.209120 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.205885 23198 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.211346 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.184258 23189 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20251214 02:59:26.204757 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.201079 23185 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.226992 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.204954 23109 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.232422 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.233882 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.231805 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 59815.7us
I20251214 02:59:26.236068 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.230309 23198 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.209446 23112 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.236671 21275 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:26.205881 23205 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.237723 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:26.238620 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 46757.5us
I20251214 02:59:26.225538 23193 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:26.239959 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.233264 21173 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.243157 21985 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:26.243360 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.245656 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.232465 23109 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.247468 21985 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:26.239204 23179 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.250587 23211 raft_consensus.cc:2804] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.250976 23211 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.251317 23211 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:26.243103 21247 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.529s (SENT) [suppressed 10 similar messages]
W20251214 02:59:26.248994 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45366.5us
I20251214 02:59:26.256783 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.257892 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.257524 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.239610 23205 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.260214 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.260710 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.262142 21173 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.263764 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.264914 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.266470 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.266644 21099 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.254094 23109 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.249310 23204 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:26.253546 23179 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.282235 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.266013 23186 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:26.262332 23205 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.288496 23213 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:26.289011 23213 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.289404 23213 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.290580 23217 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.291054 23217 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.291494 23217 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.257501 23190 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.302649 21248 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.306682 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.307125 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.294867 23186 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.262465 23188 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.281886 23204 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.315807 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.284085 23179 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:26.322484 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.323483 21132 raft_consensus.cc:2393] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.324999 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.330322 21124 raft_consensus.cc:2393] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.332875 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.313572 23186 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:26.335705 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:26.317710 23204 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:26.342592 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.343483 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.322408 23165 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.398211 23165 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.399029 23165 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.384330 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.385133 23214 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.323019 23181 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20251214 02:59:26.336710 23207 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:26.384255 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.402812 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.403345 21099 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:26.414975 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 171908us
I20251214 02:59:26.420338 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.420831 21279 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:26.400295 23179 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:26.423183 21203 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:26.423591 23213 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.400087 23205 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.425554 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.426590 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.431627 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 113848us
I20251214 02:59:26.434268 21275 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:26.423487 23179 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.442018 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.434870 21275 raft_consensus.cc:606] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:26.442610 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.443152 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.441131 23213 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.433809 23175 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:26.450973 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.452627 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.424185 23177 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:26.454499 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 139982us
I20251214 02:59:26.445451 23187 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:26.468639 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.469511 23162 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.469914 23162 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.470270 23162 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.471813 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.472379 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.472843 21279 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
W20251214 02:59:26.470275 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 166929us
I20251214 02:59:26.473686 22017 raft_consensus.cc:1240] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:26.475041 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:26.466611 23190 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.476830 23177 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.457156 21129 raft_consensus.cc:1240] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.466378 23188 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.442742 23211 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.482494 21205 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:26.479315 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.490954 23089 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.515091 23198 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.517155 23089 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.517539 23211 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.498768 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.518247 23198 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.519021 21128 raft_consensus.cc:2393] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.500065 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.512046 23109 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.520509 23205 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.520715 21133 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.494889 23226 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:26.511312 23210 consensus_queue.cc:1059] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20251214 02:59:26.522233 23226 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.492740 23165 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.499414 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.492425 23199 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.523639 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.519236 23129 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.498109 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.524391 23199 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.490861 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:26.525017 21274 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.525558 23129 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:26.526696 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47457.3us
I20251214 02:59:26.521420 23109 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.472632 23186 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.497485 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.493152 23192 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.488648 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:26.498667 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.491931 21201 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251214 02:59:26.526590 23204 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.534226 23179 raft_consensus.cc:3055] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:26.507457 23187 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.497222 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.534605 23179 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:26.536132 23179 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.511093 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.539016 23179 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.516706 23195 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.521803 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.501550 23217 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.522461 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.457239 23183 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.544698 23217 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.545302 23183 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.495831 23112 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.547161 23112 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:26.528672 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 125417us
I20251214 02:59:26.519271 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.549031 21276 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:26.535250 23186 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.455808 23167 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.528302 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.537263 23187 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.535346 23204 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:26.537633 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49072.9us
I20251214 02:59:26.537024 23192 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.577369 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.554948 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.575040 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:26.564121 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 283445us
I20251214 02:59:26.585978 22016 raft_consensus.cc:1240] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.587175 21131 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251214 02:59:26.587734 23185 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.588044 23188 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:26.588758 21207 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.589039 23187 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:26.589483 23187 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.590407 21208 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:26.590528 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.587216 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.591322 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.591877 23217 consensus_queue.cc:1059] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:26.592808 23186 raft_consensus.cc:3055] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:26.593192 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.593493 21130 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.593850 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.595679 23212 consensus_queue.cc:1059] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:26.595994 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.596328 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.596683 23212 raft_consensus.cc:3055] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:26.597034 23212 raft_consensus.cc:740] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:26.593293 23173 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:26.597355 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.597875 23173 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:26.598110 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.599009 21126 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:26.599327 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.593142 23186 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.601365 23186 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.602241 23185 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.602496 23186 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.589412 23188 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:26.597762 23212 consensus_queue.cc:260] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.600517 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.605785 21124 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:26.605448 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:26.510540 23162 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.549839 21125 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.640988 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.526875 23226 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.655658 23201 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:26.657634 23167 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.617552 23171 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:26.658960 23171 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:26.659335 23171 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.606982 23173 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.663261 23173 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.665467 23165 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:26.607113 23188 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.674774 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.675200 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 146976us
I20251214 02:59:26.675666 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.676630 21129 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.678053 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.678207 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.667344 23173 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.687103 21205 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
W20251214 02:59:26.668128 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 188132us
I20251214 02:59:26.681727 21201 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:26.688093 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:26.688825 23213 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.688637 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.667824 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.692315 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.689644 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.694326 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.698529 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.698784 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.689266 23213 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.700044 23213 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.700306 23209 consensus_queue.cc:1059] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:26.700716 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.701740 23183 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:26.679884 23188 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.702706 23183 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:26.702894 23188 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.703151 23164 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.703503 23209 raft_consensus.cc:3055] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:26.700044 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.703951 23209 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.704238 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.704845 23167 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:26.704713 23209 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.705930 23209 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:26.706344 23167 raft_consensus.cc:3055] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:26.706738 23167 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:26.706867 23162 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:26.707443 23183 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.707619 23167 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.708231 23183 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.709144 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:26.709396 23183 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.703573 23164 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:26.696455 23192 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.711390 23192 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:26.711724 23192 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.714110 21125 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:26.710978 23164 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.718441 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.720655 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.722839 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.723270 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:26.723600 21248 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.721457 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:26.725288 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:26.725584 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:26.713846 23167 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.726899 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:26.734504 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.727957 23195 raft_consensus.cc:993] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:26.728614 23224 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.730521 23196 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.742240 23196 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.742647 23196 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.747387 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:26.750726 23205 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:26.750792 23195 raft_consensus.cc:1081] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:26.751130 23205 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:26.751544 23205 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.752275 23171 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.727994 22015 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:26.754164 23171 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.759194 21276 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.762252 21127 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:26.766952 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.762804 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 297) took 3384 ms. Trace:
I20251214 02:59:26.718688 23212 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.752764 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 261) took 3409 ms. Trace:
I20251214 02:59:26.762933 21201 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:26.765076 23217 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:26.771147 22015 raft_consensus.cc:1240] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
W20251214 02:59:26.768097 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 46708us
I20251214 02:59:26.769520 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.770217 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.771689 21124 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:26.767500 21029 rpcz_store.cc:276] 1214 02:59:23.378166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:23.378301 (+ 135us) service_pool.cc:225] Handling call
1214 02:59:23.416980 (+ 38679us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b54000cadd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6714d2513c 00007f6714e1f44e 00007f671331d4a4 00007f671331d432 00007f671331d361 00007f671331d2dc 00007f671331c592 00007f670d2dd55c 00007f670d2dc996
1214 02:59:26.762733 (+3345753us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2413,"reported_tablets":3,"spinlock_wait_cycles":42781440}
I20251214 02:59:26.767364 21200 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:26.774165 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:26.784562 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:26.785712 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.785956 23230 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:26.787658 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:26.788391 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.788688 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.789453 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.789642 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.786325 23230 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.790836 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.791896 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.793331 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.794056 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.768054 21173 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.774111 21278 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.790917 23230 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.799108 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.800580 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.800716 23210 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:26.798485 23112 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.807281 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:26.807838 21279 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:26.806326 21278 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:26.775220 23205 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.812433 23205 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.815945 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.771975 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.817662 21985 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.823793 23165 raft_consensus.cc:993] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:26.824271 23165 raft_consensus.cc:1081] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Signalling peer 8de0cf3ca4bf4083bc494a9af0e49342 to start an election
I20251214 02:59:26.819422 21096 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.825687 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.826735 23209 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.826918 21245 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.768258 21028 rpcz_store.cc:276] 1214 02:59:23.343679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:23.343842 (+ 163us) service_pool.cc:225] Handling call
1214 02:59:23.410963 (+ 67121us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b54000cadd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6714d2513c 00007f6714e1f44e 00007f671331d4a4 00007f671331d432 00007f671331d361 00007f671331d2dc 00007f671331c592 00007f670d2dd55c 00007f670d2dc996
1214 02:59:26.752698 (+3341735us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:26.212375 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:26.300186 (+ 87811us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:26.301600 (+ 1414us) write_op.cc:435] Acquired schema lock
1214 02:59:26.302611 (+ 1011us) tablet.cc:660] Decoding operations
1214 02:59:26.336833 (+ 34222us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:26.398886 (+ 62053us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:26.398928 (+ 42us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:26.399133 (+ 205us) tablet.cc:699] Row locks acquired
1214 02:59:26.399151 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:26.399305 (+ 154us) write_op.cc:270] Start()
1214 02:59:26.399413 (+ 108us) write_op.cc:276] Timestamp: P: 1765681166399257 usec, L: 0
1214 02:59:26.399431 (+ 18us) op_driver.cc:348] REPLICATION: starting
1214 02:59:26.400091 (+ 660us) log.cc:844] Serialized 541 byte log entry
1214 02:59:26.676584 (+276493us) op_driver.cc:464] REPLICATION: finished
1214 02:59:26.741372 (+ 64788us) write_op.cc:301] APPLY: starting
1214 02:59:26.741450 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:26.741541 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:26.741656 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:26.741847 (+ 191us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:26.742048 (+ 201us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:26.742089 (+ 41us) write_op.cc:312] APPLY: finished
1214 02:59:26.742283 (+ 194us) log.cc:844] Serialized 23 byte log entry
1214 02:59:26.742433 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:26.742512 (+ 79us) write_op.cc:454] Released schema lock
1214 02:59:26.742547 (+ 35us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:26.742883 (+ 336us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:26.742911 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":437978,"reported_tablets":5,"spinlock_wait_cycles":143126400,"child_traces":[["op",{"apply.queue_time_us":64562,"apply.run_cpu_time_us":1510,"apply.run_wall_time_us":1510,"num_ops":1,"prepare.queue_time_us":168998,"prepare.run_cpu_time_us":17017,"prepare.run_wall_time_us":282114,"raft.queue_time_us":205020,"raft.run_cpu_time_us":9641,"raft.run_wall_time_us":44990,"replication_time_us":277017,"thread_start_us":422855,"threads_started":4,"wal-append.queue_time_us":68720}]]}
I20251214 02:59:26.828281 23195 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:26.829286 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:26.831732 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:26.834820 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.835667 21202 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:26.836159 21202 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:26.836475 21202 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.758548 23192 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.839932 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.840687 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.842084 21125 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:26.848201 21985 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:26.851889 21279 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 02:59:26.854303 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:26.861301 19986 debug-util.cc:398] Leaking SignalData structure 0x7b080000f240 after lost signal to thread 21173
I20251214 02:59:26.862797 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:26.864259 21128 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:26.864270 21096 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:26.868301 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 52444.5us
W20251214 02:59:26.869344 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.869833 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.899570 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:26.903688 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 55597.6us
I20251214 02:59:26.904742 21985 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:26.907196 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.908207 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.910491 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.840013 23192 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:26.911268 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.911520 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.939925 21202 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:26.928092 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:26.862645 23167 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:26.949113 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:26.950115 21128 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 6: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:26.950371 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.901925 23212 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:26.951551 21202 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.952373 23215 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:26.942430 23211 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:26.954855 23211 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:26.955255 23211 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:26.955643 23196 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.957521 23196 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.905135 23112 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:26.905417 23109 consensus_queue.cc:1059] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20251214 02:59:26.963413 23109 raft_consensus.cc:3055] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:26.963742 23109 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:26.964468 23109 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.965653 23109 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:26.965831 23204 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.966598 23177 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:26.967206 23177 consensus_queue.cc:1059] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:26.968815 23177 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:26.969163 23177 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:26.969234 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.971395 23214 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:26.972455 21126 raft_consensus.cc:2410] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20251214 02:59:26.973093 23214 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:26.974524 23187 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.970306 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:26.976558 23187 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:26.977196 23205 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.978345 23193 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:26.969933 23177 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.979436 23177 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.979938 23179 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:26.981091 23112 raft_consensus.cc:3055] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:26.981463 23112 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:26.982280 23112 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.905908 23230 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.985483 23193 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:26.986903 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.987823 23208 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:26.988292 23208 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:26.989209 23208 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:26.991444 22015 raft_consensus.cc:1240] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:26.945392 23213 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:26.992708 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:26.994792 21199 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.000207 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.000474 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.000988 21128 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.004323 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.008948 21128 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:27.012449 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.013161 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.013725 21278 raft_consensus.cc:3055] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:27.014232 21278 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.015975 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.017165 21125 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:26.904641 23186 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.023882 21205 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:27.025058 21133 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.018725 21278 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.028345 21278 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:26.985559 23230 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:27.031989 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.033211 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.012095 22016 raft_consensus.cc:1240] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:27.037849 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.038868 21173 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:26.902590 23164 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.018607 23193 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:27.042944 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.047192 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.021098 23187 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:27.049811 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:27.057017 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 212033us
I20251214 02:59:27.057936 21986 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:27.062852 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.064200 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.066953 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:27.069217 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.070679 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:27.071997 21245 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:27.073760 21278 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:27.074294 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.074007 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
W20251214 02:59:27.076351 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 171707us
I20251214 02:59:27.076562 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.076679 21245 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:27.077142 21985 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.080312 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.080480 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.065344 23089 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:27.083199 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.056190 23210 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.089124 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.090333 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.030007 23226 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.086956 22017 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.087324 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.087646 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.075691 23221 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.075564 23214 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:27.109184 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.065582 23193 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:27.073971 23230 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:27.085240 21202 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.095755 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.073328 23185 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:27.100554 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.077634 22014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd"
dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
from {username='slave'} at 127.0.0.1:44272
I20251214 02:59:27.101611 21277 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
W20251214 02:59:27.102691 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.095736 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.081691 23162 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.109277 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.085759 23165 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:27.073534 23207 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:27.121271 21130 raft_consensus.cc:2393] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:27.121768 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.121963 23230 raft_consensus.cc:697] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:27.125005 21278 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:27.124437 21275 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:27.135383 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.119148 23214 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:27.125555 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:27.177317 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:27.121886 23188 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:27.178406 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 109767us
I20251214 02:59:27.127593 23196 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.127348 23164 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:27.185114 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49797.6us
I20251214 02:59:27.124009 23236 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.182150 23213 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:27.125593 23224 consensus_queue.cc:1059] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20251214 02:59:27.195516 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:27.200269 23129 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:27.201973 23210 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:27.202487 23210 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:27.192569 22014 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:27.203024 22014 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.203718 21129 raft_consensus.cc:2393] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:27.203316 23210 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.207093 23214 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
W20251214 02:59:27.210397 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.218919 23205 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.192648 23193 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.223673 23226 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:27.230849 23211 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.127445 23230 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.191668 23238 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:27.236927 23220 consensus_queue.cc:1059] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:27.252066 22014 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.255558 23204 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251214 02:59:27.269966 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.324031 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.291523 22017 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:27.327734 23237 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:27.328529 23237 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251214 02:59:27.304227 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 183278us
I20251214 02:59:27.329418 22013 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:27.330287 23211 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:27.309900 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.331812 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.330435 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.333374 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.333634 21986 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.334978 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.367954 23164 raft_consensus.cc:3055] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.381415 23164 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:27.382344 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 151655us
I20251214 02:59:27.382208 23164 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.382702 23204 raft_consensus.cc:2804] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:27.383253 23204 raft_consensus.cc:697] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:27.383370 23164 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.383770 21245 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:27.384159 23204 consensus_queue.cc:237] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.384573 23165 consensus_queue.cc:1059] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251214 02:59:27.385129 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.385676 23192 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:27.386631 22017 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.387746 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:27.396584 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 60294us
I20251214 02:59:27.417236 23195 consensus_queue.cc:1059] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20251214 02:59:27.420480 23196 consensus_queue.cc:1059] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20251214 02:59:27.420993 23195 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.421411 23195 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.417925 23213 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20251214 02:59:27.422225 23195 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.423231 23195 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:27.424928 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.431735 23236 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.432994 23196 raft_consensus.cc:3055] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:27.433367 23196 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:27.417693 22014 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:27.412741 23187 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:27.419656 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.419874 23186 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:27.398022 23192 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.397289 23237 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.390887 23209 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.437558 23213 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:27.437184 23162 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:27.441180 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:27.444206 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.453100 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:27.454700 23196 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.455703 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.457043 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:27.455624 23171 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20251214 02:59:27.444762 23112 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.472111 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:27.460507 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 130152us
I20251214 02:59:27.462399 22012 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:27.473377 21203 raft_consensus.cc:2393] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:27.464116 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.465461 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.474236 21098 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:27.468371 21200 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:27.475411 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.469827 21128 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.471410 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.473804 22017 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:27.483772 21199 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.479700 23167 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.479538 23179 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.490772 23192 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.493804 23199 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.494210 23199 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.472718 23162 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.495080 23199 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.496243 23199 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.497299 23242 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:27.482044 23209 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:27.500658 23209 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:27.501029 23209 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.503065 23109 raft_consensus.cc:3055] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.503437 23109 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:27.504137 23109 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.506386 23196 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.507123 23162 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:27.474645 23187 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:27.465958 23213 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:27.510756 21276 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:27.512131 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.517493 21128 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:27.512358 21029 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 3 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:27.521364 21098 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.529524 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:27.533869 23242 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:27.534368 23237 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:27.535458 23237 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:27.531132 21273 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:27.535900 23237 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
W20251214 02:59:27.538496 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.538945 23237 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.541527 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:27.542647 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.524380 23183 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:27.546463 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:27.547343 23173 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:27.547719 23173 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:27.545522 23242 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:27.547744 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.557433 23224 raft_consensus.cc:3055] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:27.557801 23224 raft_consensus.cc:740] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.540983 23193 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251214 02:59:27.524000 23164 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20251214 02:59:27.558531 23224 consensus_queue.cc:260] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.540891 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:27.559767 23224 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.558527 23109 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.561401 23165 raft_consensus.cc:3055] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:27.559024 23242 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:27.561764 23165 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.548050 23173 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:27.551079 23187 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.551276 23213 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.563907 23213 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:27.566637 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.544288 23217 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:27.606222 23217 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:27.606691 23217 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:27.569401 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.609474 21986 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:27.610185 23230 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:27.570286 21202 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:27.610813 23230 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:27.567601 23165 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.579442 23209 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.612232 23165 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:27.613680 23209 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:27.588367 21278 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:27.591181 23193 raft_consensus.cc:3055] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:27.789233 23193 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:27.605007 23245 log.cc:927] Time spent T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.078s user 0.004s sys 0.000s
W20251214 02:59:27.746100 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:27.752590 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.760720 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.790293 23193 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.792829 23193 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:27.794282 21030 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:27.800829 23186 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:27.801390 23186 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.803781 23186 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:27.764501 23173 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:27.752818 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 196996us
I20251214 02:59:27.780541 23109 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 02:59:27.784102 21248 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.784456 23162 consensus_queue.cc:1059] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:27.592191 22014 raft_consensus.cc:1240] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:27.814415 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:27.815961 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:27.862396 22017 raft_consensus.cc:2393] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:27.806269 23201 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:27.827782 21130 raft_consensus.cc:1240] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:27.829892 21124 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:27.863579 21096 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:27.830405 21127 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20251214 02:59:27.864372 21127 pending_rounds.cc:85] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 5
I20251214 02:59:27.834797 23217 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.850210 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.857388 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:27.864626 21127 pending_rounds.cc:107] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.6
I20251214 02:59:27.884045 23210 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:27.898694 21127 raft_consensus.cc:2917] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Aborting config change with OpId 4.6: Aborted: Op aborted by new leader
I20251214 02:59:27.899125 21127 raft_consensus.cc:2889] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:27.899085 23210 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.900573 21202 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:27.887212 21275 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:59:27.902055 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 51964.8us
I20251214 02:59:27.902810 21098 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.902953 23242 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:27.903594 23242 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:27.903952 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57314.6us
I20251214 02:59:27.905416 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:27.909873 21202 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:27.913515 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.914304 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:27.873194 23129 raft_consensus.cc:3055] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:27.916615 21098 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:27.930055 21127 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251214 02:59:27.874750 23209 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.932358 21128 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:27.934871 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.937449 23173 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:27.895463 23220 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20251214 02:59:27.896104 23224 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20251214 02:59:27.938350 23199 raft_consensus.cc:3055] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.952324 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.952502 23162 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:27.952610 23199 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.952925 23162 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:27.953540 23199 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.953738 23162 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:27.883961 23179 consensus_queue.cc:1059] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20251214 02:59:27.943584 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:27.919134 23109 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:27.918942 23186 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20251214 02:59:27.944975 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:27.915045 23228 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20251214 02:59:27.909822 23242 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:27.949414 21275 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:27.955192 23177 consensus_queue.cc:1059] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20251214 02:59:27.943171 23210 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:27.955076 23162 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.959749 23228 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:27.954697 23199 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:27.950543 23209 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:27.956260 23186 raft_consensus.cc:3055] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:27.963141 23186 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:27.963903 23186 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:27.965209 23186 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:27.949966 23217 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:27.968559 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:27.975888 21202 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:27.979252 21124 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:27.980646 23237 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:27.980767 23228 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:27.918861 23183 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:27.883901 23201 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:27.994789 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 63993.4us
I20251214 02:59:27.996227 23112 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:27.997397 21279 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 02:59:27.932531 23129 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:27.999226 21279 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251214 02:59:27.998901 23129 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.000586 23129 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:28.012528 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.018141 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.018227 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:28.019495 21274 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:28.021008 23201 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.021675 21986 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.022878 23201 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:28.028417 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 111875us
I20251214 02:59:28.028610 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.041404 23187 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:28.044780 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:28.049402 21098 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:28.050292 21248 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:28.056774 23220 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:28.057758 23220 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:28.058105 23220 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:28.058396 21247 leader_election.cc:336] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.529s (SENT)
I20251214 02:59:28.059100 21202 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:28.058883 23220 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.060266 23220 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:28.050661 21275 raft_consensus.cc:3060] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:28.042013 23187 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.060652 23224 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:28.063618 23187 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.048097 23177 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20251214 02:59:28.064540 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:28.054637 23179 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:28.065141 22014 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:28.066416 21171 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:28.049614 23183 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.068527 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:28.075564 21248 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:28.079138 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:28.083381 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.084187 21278 raft_consensus.cc:2393] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:28.086431 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.087013 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.087646 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.089658 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.096736 21278 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:28.098737 21276 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:28.098981 21280 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:28.101274 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.103313 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.103259 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.105598 21127 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:28.118916 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.120337 21282 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251214 02:59:28.079248 23179 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:28.121088 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.123517 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.123636 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.077503 23257 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:28.098481 23183 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:28.133211 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.127655 21277 raft_consensus.cc:1813] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Handling vote request from an unknown peer 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.137916 21277 raft_consensus.cc:2410] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251214 02:59:28.128410 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:28.133978 21273 raft_consensus.cc:2393] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:28.139583 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.126202 23179 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:28.170662 23179 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:28.140777 23192 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:28.137847 23214 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20251214 02:59:28.133154 23183 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:28.147660 23109 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.147472 23224 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.134112 23206 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20251214 02:59:28.211205 21275 raft_consensus.cc:1275] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:28.191363 23183 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:28.142112 23208 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20251214 02:59:28.228781 23208 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:28.229171 23208 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:28.229694 21281 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:28.217339 21273 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:28.213078 23109 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.232075 23109 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.228574 23230 consensus_queue.cc:1048] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:28.234519 23230 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:28.235512 23183 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.237457 21201 raft_consensus.cc:1240] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:28.242350 23230 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:28.244513 23230 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.243065 23109 consensus_queue.cc:1059] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20251214 02:59:28.245822 23109 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:28.246136 23109 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:28.246886 23109 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:28.247732 23109 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:28.248322 23230 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.248543 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.249313 21275 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:59:28.243062 23258 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20251214 02:59:28.249454 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:28.251392 23183 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.229924 23208 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.252179 21986 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:28.253000 23230 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:28.252251 23208 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:28.254948 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:28.300756 23179 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.307564 23179 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.305721 23230 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.308688 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:28.317564 22014 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:28.319337 22017 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20251214 02:59:28.319748 22017 pending_rounds.cc:85] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Aborting all ops after (but not including) 5
I20251214 02:59:28.319970 22017 pending_rounds.cc:107] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.6
I20251214 02:59:28.320195 22017 raft_consensus.cc:2917] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Aborting config change with OpId 4.6: Aborted: Op aborted by new leader
I20251214 02:59:28.320542 22017 raft_consensus.cc:2889] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:28.322805 21171 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:28.324712 23187 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:28.325465 23112 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:28.350544 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:28.355252 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 217329us
I20251214 02:59:28.359709 22017 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:28.359755 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 220263us
I20251214 02:59:28.360553 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.361614 21247 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.363629 21127 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:28.366257 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:28.366703 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.369397 21247 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.355439 23208 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.358361 23112 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:28.372162 21247 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.367182 22017 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:28.376885 21282 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:28.397167 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.370646 23164 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.398824 21127 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:28.405236 21127 raft_consensus.cc:606] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:28.405579 21127 raft_consensus.cc:614] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:28.399499 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.372910 23162 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:28.370334 23192 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:28.417093 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.400213 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.400694 21279 raft_consensus.cc:2393] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:28.402472 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.373030 23195 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:28.421038 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.421408 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.422039 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.406828 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.410120 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.395037 23263 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.424036 21126 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:28.425988 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.418309 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.463829 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.464349 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.464133 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.464872 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.413403 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:28.423169 21124 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:28.382690 23199 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.465593 21132 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:28.483474 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:28.494060 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 80722.1us
I20251214 02:59:28.494982 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:28.496672 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.496876 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.414340 23173 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:28.382227 23208 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.503091 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.493269 21127 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:28.462795 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.506282 23274 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20251214 02:59:28.509095 23109 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:28.510057 23109 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:28.511224 23208 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.510783 23112 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.512745 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:28.513325 22017 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:28.514583 21171 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:28.423849 23129 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
W20251214 02:59:28.508323 23282 log.cc:927] Time spent T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20251214 02:59:28.467943 23263 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.525305 23281 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
W20251214 02:59:28.559854 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 141172us
I20251214 02:59:28.562083 23281 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:28.516510 23173 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.583899 23193 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:28.559984 21247 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.586452 21133 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:28.595708 21132 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:28.599368 22017 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:28.606748 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:28.607185 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 47293.3us
I20251214 02:59:28.581455 23257 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:28.607419 21132 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:28.622913 23257 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:28.629855 23257 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:28.635924 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 232010us
I20251214 02:59:28.589279 23162 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.636682 21096 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:28.609272 23171 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:28.660904 21199 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:28.598097 23263 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.659494 21279 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:59:28.640362 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:28.678709 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:28.660245 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:28.679352 21206 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
W20251214 02:59:28.638865 23279 log.cc:927] Time spent T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20251214 02:59:28.638001 23187 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
W20251214 02:59:28.637269 23273 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20251214 02:59:28.639915 23183 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:28.708942 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:28.711174 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.755451 21279 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:59:28.723690 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 102524us
I20251214 02:59:28.640354 23162 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.708279 23173 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.732364 23183 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:28.784447 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:28.785321 22017 raft_consensus.cc:2393] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:28.704908 23199 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20251214 02:59:28.761509 23257 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:28.788327 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 293460us
I20251214 02:59:28.789173 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:28.789131 21099 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:28.793488 23162 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:28.793793 23253 log.cc:927] Time spent T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251214 02:59:28.795022 23263 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:28.795492 23263 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:28.795841 23263 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:28.796770 23208 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.797312 23208 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:28.799991 23208 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.800896 23294 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.696355 23281 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.812119 23294 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.812552 23281 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:28.814076 23294 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.814172 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:28.785153 23183 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:28.818933 23263 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:28.820118 23274 consensus_queue.cc:1175] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "668ad4f30bd649c88a034bbab9f13b20" responder_term: 5 status { last_received { term: 5 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 5 index: 7 } }
I20251214 02:59:28.789268 23173 consensus_queue.cc:1059] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20251214 02:59:28.821455 23263 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.821547 23173 raft_consensus.cc:3055] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:28.821921 23173 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:28.822757 23173 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.823884 23173 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:28.825510 23183 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.808848 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:28.811239 23293 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.805824 23278 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:28.809448 23199 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.920212 22017 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:28.923488 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:28.924152 21274 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:28.923305 23195 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.926465 23195 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.925978 21245 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:28.929435 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:28.930706 21247 leader_election.cc:336] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.542s (SENT)
I20251214 02:59:28.931185 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:28.936265 23289 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:28.941535 23195 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:28.941946 23295 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:28.942343 23295 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:28.942718 23295 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:28.942855 23257 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.926600 23221 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.950887 23183 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.946502 23162 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.953289 23162 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.955242 23162 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:28.948167 23221 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.959638 23221 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.962868 23213 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:28.963402 23213 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.965174 23213 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:28.966960 23291 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:28.967496 23291 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.971635 23295 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.973498 23295 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.927659 23199 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:28.975571 23199 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.924407 23293 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:28.977500 23291 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:28.978086 23293 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:29.010792 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 219699us
W20251214 02:59:29.009732 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 220411us
W20251214 02:59:29.013859 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:29.014940 21098 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:29.016387 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.018316 21128 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:29.023098 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.023994 21128 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.024071 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.024959 21130 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:29.025317 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.025977 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.026165 21128 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:29.027328 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.028339 21124 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.029830 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:29.031029 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.031962 21127 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.035224 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.035892 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.036036 21203 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:29.043874 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.044612 22012 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.046865 21130 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:29.051172 23109 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:29.056975 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.060563 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.060798 23208 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:29.061369 23208 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.061545 21128 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
W20251214 02:59:29.063138 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 48574us
I20251214 02:59:29.064311 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.067404 23247 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:29.067727 23173 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:29.068454 23173 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:29.071074 23187 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:29.071520 21247 leader_election.cc:336] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.605s (SENT)
I20251214 02:59:29.072212 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.068397 23247 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.075057 23195 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.067149 23288 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251214 02:59:29.062330 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 53076.9us
I20251214 02:59:29.080560 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.081424 22018 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:29.061792 23208 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.077992 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.086198 22017 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:29.070000 23173 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.087306 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.088232 22016 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.090209 23187 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:29.091714 21247 leader_election.cc:336] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.539s (SENT)
I20251214 02:59:29.092236 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.093437 23187 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.097764 21098 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.099151 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.099392 23281 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.100064 21125 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:29.105110 23215 consensus_queue.cc:1059] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:29.110487 23247 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.110957 21245 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.111227 23240 raft_consensus.cc:3055] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:29.111807 23240 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:29.113006 23240 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.117897 21127 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:29.122334 22014 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:29.123862 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.124789 21273 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:29.130895 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.131737 21282 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:29.132535 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.133586 21281 raft_consensus.cc:2393] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:29.138072 21098 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.142266 23195 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:29.142971 23224 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.143676 23224 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:29.144068 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 52479.7us
I20251214 02:59:29.144701 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.145987 23224 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.146027 23304 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:29.146678 23304 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:29.147148 23304 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:29.148339 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 299) took 1357 ms. Trace:
I20251214 02:59:29.149646 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 154) took 1661 ms. Trace:
I20251214 02:59:29.150539 21028 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:29.155604 21203 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:29.160854 22012 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:29.150295 23171 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:29.180467 22017 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:29.190778 23278 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:29.191368 23278 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.191722 23278 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.194801 23274 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.195756 23240 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.173177 23171 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.243171 23171 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:29.244726 23310 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.070s user 0.003s sys 0.000s
I20251214 02:59:29.150087 21029 rpcz_store.cc:276] 1214 02:59:27.488234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:27.488353 (+ 119us) service_pool.cc:225] Handling call
1214 02:59:29.149479 (+1661126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:27.897755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:27.898293 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:27.898328 (+ 35us) write_op.cc:435] Acquired schema lock
1214 02:59:27.898344 (+ 16us) tablet.cc:660] Decoding operations
1214 02:59:27.898666 (+ 322us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:27.898739 (+ 73us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:27.898765 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:27.898944 (+ 179us) tablet.cc:699] Row locks acquired
1214 02:59:27.898958 (+ 14us) write_op.cc:260] PREPARE: finished
1214 02:59:27.899091 (+ 133us) write_op.cc:270] Start()
1214 02:59:27.899192 (+ 101us) write_op.cc:276] Timestamp: P: 1765681167899045 usec, L: 0
1214 02:59:27.899210 (+ 18us) op_driver.cc:348] REPLICATION: starting
1214 02:59:27.899822 (+ 612us) log.cc:844] Serialized 536 byte log entry
1214 02:59:28.171045 (+271223us) op_driver.cc:464] REPLICATION: finished
1214 02:59:28.330411 (+159366us) write_op.cc:301] APPLY: starting
1214 02:59:28.330498 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:28.330623 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:28.330738 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:28.330962 (+ 224us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:28.331145 (+ 183us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:28.331197 (+ 52us) write_op.cc:312] APPLY: finished
1214 02:59:28.354724 (+ 23527us) log.cc:844] Serialized 23 byte log entry
1214 02:59:28.359793 (+ 5069us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:28.371705 (+ 11912us) write_op.cc:454] Released schema lock
1214 02:59:28.374187 (+ 2482us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:28.401842 (+ 27655us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:28.414013 (+ 12171us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20247,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":159135,"apply.run_cpu_time_us":1788,"apply.run_wall_time_us":62474,"mutex_wait_us":10806,"num_ops":1,"prepare.queue_time_us":374409,"prepare.run_cpu_time_us":8366,"prepare.run_wall_time_us":126061,"raft.queue_time_us":202265,"raft.run_cpu_time_us":17165,"raft.run_wall_time_us":187636,"replication_time_us":271685,"thread_start_us":420792,"threads_started":4,"wal-append.queue_time_us":118440}]]}
I20251214 02:59:29.251039 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.251713 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.252421 21202 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:29.201460 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.202026 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.204141 23204 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.265492 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:29.205731 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 67856us
I20251214 02:59:29.210425 21099 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:29.212627 23304 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.216706 23213 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.226725 23295 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.232180 23230 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:29.233275 21247 leader_election.cc:336] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.632s (SENT)
I20251214 02:59:29.240100 23215 raft_consensus.cc:1064] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 02:59:29.148772 21030 rpcz_store.cc:276] 1214 02:59:27.791011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:27.791181 (+ 170us) service_pool.cc:225] Handling call
1214 02:59:29.148110 (+1356929us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:28.023950 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:28.024466 (+ 516us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:28.024493 (+ 27us) write_op.cc:435] Acquired schema lock
1214 02:59:28.024509 (+ 16us) tablet.cc:660] Decoding operations
1214 02:59:28.024832 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:28.024882 (+ 50us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:28.024915 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:28.025101 (+ 186us) tablet.cc:699] Row locks acquired
1214 02:59:28.025114 (+ 13us) write_op.cc:260] PREPARE: finished
1214 02:59:28.025255 (+ 141us) write_op.cc:270] Start()
1214 02:59:28.027685 (+ 2430us) write_op.cc:276] Timestamp: P: 1765681168025208 usec, L: 0
1214 02:59:28.027715 (+ 30us) op_driver.cc:348] REPLICATION: starting
1214 02:59:28.066680 (+ 38965us) log.cc:844] Serialized 536 byte log entry
1214 02:59:28.302989 (+236309us) op_driver.cc:464] REPLICATION: finished
1214 02:59:28.461512 (+158523us) write_op.cc:301] APPLY: starting
1214 02:59:28.467688 (+ 6176us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:28.497849 (+ 30161us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:28.525113 (+ 27264us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:28.540252 (+ 15139us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:28.559094 (+ 18842us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:28.586139 (+ 27045us) write_op.cc:312] APPLY: finished
1214 02:59:28.622348 (+ 36209us) log.cc:844] Serialized 23 byte log entry
1214 02:59:28.629568 (+ 7220us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:28.629662 (+ 94us) write_op.cc:454] Released schema lock
1214 02:59:28.629707 (+ 45us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:28.630040 (+ 333us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:28.630067 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":512034,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":121047,"apply.run_cpu_time_us":2030,"apply.run_wall_time_us":202730,"num_ops":1,"prepare.queue_time_us":223505,"prepare.run_cpu_time_us":2595,"prepare.run_wall_time_us":42996,"replication_time_us":273194,"thread_start_us":110800,"threads_started":1}]]}
I20251214 02:59:29.204921 21096 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.172932 22016 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:29.272759 22017 raft_consensus.cc:2393] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:29.273281 22012 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:29.273476 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.279250 21245 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.234385 23208 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.282428 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:29.282581 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 120092us
W20251214 02:59:29.285969 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 75723.9us
W20251214 02:59:29.290079 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 91871.6us
I20251214 02:59:29.278920 23230 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.292155 23247 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 02:59:29.293709 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 257963us
I20251214 02:59:29.292040 23274 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:29.280290 23195 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.291805 23292 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:29.287396 23213 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.300029 23213 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.300395 23195 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.301184 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.301610 23274 raft_consensus.cc:697] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:29.301887 21201 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:29.302479 23230 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:29.282343 23204 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.302770 23274 consensus_queue.cc:237] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.305128 23204 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:29.305274 23164 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:29.305761 23164 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.282872 23295 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:29.300879 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 67649.5us
I20251214 02:59:29.307654 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.308760 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.309092 23292 raft_consensus.cc:697] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:29.309505 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.310083 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.310256 21127 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 02:59:29.311036 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.311244 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:29.311769 21133 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:29.299762 23304 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.310341 21279 raft_consensus.cc:2393] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:29.312288 23215 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.315506 21245 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.315860 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:29.316977 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.317638 23173 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:29.318382 23263 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:29.319147 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.319823 21277 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:29.320462 21098 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.308065 23164 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.323192 23295 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.324148 21171 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.325759 21132 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:29.325484 23217 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251214 02:59:29.326853 23263 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.319087 23187 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:29.327725 23187 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.328125 23187 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.333338 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.334025 23217 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:29.334725 23217 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.337535 21125 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 1.
I20251214 02:59:29.334052 23295 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:29.338958 23208 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.339152 23295 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:29.340237 23295 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.310341 23195 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.343083 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.343521 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.327117 22016 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:29.310081 23292 consensus_queue.cc:237] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.344347 22018 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:29.343144 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.345602 22012 raft_consensus.cc:2393] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 2: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:29.346298 21245 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.345706 22019 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:29.351919 23129 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:29.352321 23129 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.352698 23129 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.352772 23210 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:29.353392 23210 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:29.354399 23210 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.354760 23257 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:29.358397 21985 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.352424 23247 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:29.360786 23247 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.361145 23247 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.370868 21245 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.326524 23240 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:29.378448 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.350461 23217 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.380968 23224 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:29.381428 23224 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.371212 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.381796 23224 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.389905 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.390321 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.394785 23238 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:29.395274 23238 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.395638 23238 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.397086 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.378037 23240 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.406821 23213 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:29.412806 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.409840 21200 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:29.414817 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.416893 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.418226 21985 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.416271 21099 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.411136 23240 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.423812 21273 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
W20251214 02:59:29.424535 21247 leader_election.cc:336] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.831s (SENT)
I20251214 02:59:29.412017 23213 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.425629 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.430344 21201 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:29.424328 23304 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:29.435846 23201 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:29.436038 23112 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.436489 23201 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:29.436611 23112 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.435900 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.438217 21126 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.427531 23274 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:29.427690 23213 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.444347 23221 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:29.444923 23221 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.446590 23201 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.450186 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:29.441721 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 121413us
I20251214 02:59:29.455107 23295 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:29.426744 23164 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.462025 22017 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEARNER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:29.462877 23164 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.463935 23263 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.464877 23220 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:29.465485 23129 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.439569 23304 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.466742 23304 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:29.467088 23171 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.472318 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:29.467680 23129 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.484266 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.485883 21274 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.457034 23274 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:29.502763 23274 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:29.492012 23173 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:29.507083 23173 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:29.507927 21098 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.508064 23173 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.499606 23171 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:29.511528 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 92081.1us
I20251214 02:59:29.433825 23278 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:29.485349 23263 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.511293 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.516615 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.516952 21201 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 5
I20251214 02:59:29.517529 22017 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 02:59:29.524775 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 94742.2us
I20251214 02:59:29.523777 21171 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.514107 23217 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.526515 23186 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:29.526693 23263 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.534917 23186 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.528144 23129 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251214 02:59:29.529385 21247 leader_election.cc:336] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.912s (SENT)
I20251214 02:59:29.496496 23112 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.533761 23274 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.540879 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.541154 23240 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.541576 21203 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:29.544647 21247 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.539058 23129 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:29.546075 23129 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:29.556216 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:29.516407 23221 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.562109 21274 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:29.563809 21126 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:29.570703 23247 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.575578 23186 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.576218 23247 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.579598 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.580664 23240 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.582403 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.583029 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.610924 22019 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.612294 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.613740 21247 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.614480 23171 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:29.614748 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:29.615216 23199 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:29.614657 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.585420 21201 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:29.585728 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:29.619624 22019 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:29.589815 23238 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.568162 23224 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.620718 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.621703 23224 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.622077 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.623334 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.623458 23187 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.623919 21279 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 3
I20251214 02:59:29.601658 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.625644 23187 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.569118 23164 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:29.626919 23238 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:29.627622 23257 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:29.628731 21248 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:29.604964 23129 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.606743 23274 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.607450 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.604966 23213 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.616060 23171 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.616231 23263 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.693159 21199 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:29.617515 21247 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.583792 23304 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.618803 23186 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:29.619733 21200 raft_consensus.cc:606] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.629228 23257 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:29.697467 23186 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.619755 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.697846 23257 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:29.604578 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 171872us
I20251214 02:59:29.679919 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.685138 23213 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.694657 21200 raft_consensus.cc:614] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251214 02:59:29.694741 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 80083.7us
I20251214 02:59:29.696786 23193 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:29.699360 21202 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.586810 23278 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.699402 21277 raft_consensus.cc:2368] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:29.706284 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.705760 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.702486 21099 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:29.722350 21126 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.679956 21274 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:29.723311 21279 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:29.701169 23171 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:29.725896 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:29.720504 23193 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.699599 23186 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.697165 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.745234 21202 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:29.744685 21278 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
W20251214 02:59:29.700182 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 71642.7us
I20251214 02:59:29.724079 23164 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:29.697005 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.749151 23242 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:29.736871 23274 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:29.755050 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.759999 22017 raft_consensus.cc:2368] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3.
I20251214 02:59:29.756891 21126 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:29.757905 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.760572 21129 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.760939 21127 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:29.761154 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.739729 23193 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.762648 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.763238 21131 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.764405 23214 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:29.764974 23214 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.766842 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.767105 23196 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.767530 21125 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.767617 23196 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.761700 21124 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.770205 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.771308 21128 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.763161 21130 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:29.776893 21131 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:29.778008 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
W20251214 02:59:29.779093 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 161658us
I20251214 02:59:29.783891 21281 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:29.784175 23304 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:29.784544 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.785228 22019 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:29.766733 23242 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.745015 23171 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:29.770192 23196 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.751592 23221 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:29.793198 21985 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.794844 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.795141 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.795680 21279 raft_consensus.cc:2393] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:29.795784 21274 raft_consensus.cc:2368] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
W20251214 02:59:29.785470 23321 log.cc:927] Time spent T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20251214 02:59:29.798681 21273 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:29.799913 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.800560 22019 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:29.801215 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.801806 21281 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.805371 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.806046 21126 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:29.758718 23112 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.769227 23193 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.781311 23274 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:29.812986 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.813500 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:29.813748 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.816061 21247 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.814045 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:29.814795 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.815403 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:29.816402 22012 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.818059 21276 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.816223 21200 raft_consensus.cc:606] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.807489 23304 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:29.823786 21199 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:29.824877 21274 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 7
I20251214 02:59:29.815533 23214 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.823156 21282 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:29.809010 23221 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:29.834874 23129 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.835132 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:29.824404 23112 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.838356 23112 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.839392 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.840029 21280 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:29.830364 23164 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:29.845487 22019 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:29.847407 21124 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:29.847467 21130 raft_consensus.cc:1240] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:29.851472 23193 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.851588 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:29.852474 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:29.853417 23205 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:29.853905 23205 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.854923 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.852761 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:29.855511 21127 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 6
I20251214 02:59:29.855965 23205 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.856926 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:29.855274 23326 log.cc:927] Time spent T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.097s user 0.002s sys 0.000s
I20251214 02:59:29.859787 22020 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:29.860616 21096 leader_election.cc:400] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3.
I20251214 02:59:29.861464 21096 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.862226 23263 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:29.862615 23263 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:29.862965 23263 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.863796 21245 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:29.865648 23224 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.866832 23242 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:29.866636 23221 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:29.870024 23274 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.852847 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.872525 21125 raft_consensus.cc:2393] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:29.872486 23263 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.876077 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.876482 23263 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:29.852424 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.880440 21133 raft_consensus.cc:2368] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:29.885601 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.886250 22016 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.887058 21128 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.887995 21275 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:29.890961 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.891580 21200 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:29.894201 21986 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.894476 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:29.897918 21127 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:29.898767 23171 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.901619 22020 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20251214 02:59:29.899725 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 101209us
I20251214 02:59:29.903213 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.904436 21247 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:29.904987 23247 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:29.905592 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:29.905781 23247 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:29.904215 23199 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:29.906482 23295 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:29.907027 23295 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:29.899180 21029 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:29.907902 23295 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.909073 21029 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 5 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:29.900810 21281 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.849781 23304 raft_consensus.cc:697] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:29.911294 23263 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:29.911535 23304 consensus_queue.cc:237] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.914029 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 262) took 2426 ms. Trace:
I20251214 02:59:29.918555 21128 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:29.863080 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.919842 21201 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:29.919760 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:29.914397 21028 rpcz_store.cc:276] 1214 02:59:27.488054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:27.513713 (+ 25659us) service_pool.cc:225] Handling call
1214 02:59:29.913956 (+2400243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:29.368485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:29.369148 (+ 663us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:29.369204 (+ 56us) write_op.cc:435] Acquired schema lock
1214 02:59:29.369221 (+ 17us) tablet.cc:660] Decoding operations
1214 02:59:29.369612 (+ 391us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:29.369697 (+ 85us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:29.369731 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:29.370032 (+ 301us) tablet.cc:699] Row locks acquired
1214 02:59:29.370046 (+ 14us) write_op.cc:260] PREPARE: finished
1214 02:59:29.370205 (+ 159us) write_op.cc:270] Start()
1214 02:59:29.370352 (+ 147us) write_op.cc:276] Timestamp: P: 1765681169370142 usec, L: 0
1214 02:59:29.370382 (+ 30us) op_driver.cc:348] REPLICATION: starting
1214 02:59:29.379332 (+ 8950us) log.cc:844] Serialized 599 byte log entry
1214 02:59:29.525837 (+146505us) op_driver.cc:464] REPLICATION: finished
1214 02:59:29.823112 (+297275us) write_op.cc:301] APPLY: starting
1214 02:59:29.844034 (+ 20922us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:29.878979 (+ 34945us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:29.879123 (+ 144us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:29.879385 (+ 262us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:29.879619 (+ 234us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:29.879669 (+ 50us) write_op.cc:312] APPLY: finished
1214 02:59:29.879928 (+ 259us) log.cc:844] Serialized 23 byte log entry
1214 02:59:29.880136 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:29.880219 (+ 83us) write_op.cc:454] Released schema lock
1214 02:59:29.880268 (+ 49us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:29.880619 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:29.880646 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":57170,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":259716,"apply.run_cpu_time_us":2002,"apply.run_wall_time_us":68520,"num_ops":1,"prepare.queue_time_us":131004,"prepare.run_cpu_time_us":2999,"prepare.run_wall_time_us":11155,"replication_time_us":122012,"thread_start_us":137404,"threads_started":2}]]}
I20251214 02:59:29.920384 21202 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:29.919206 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:29.925329 21129 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 02:59:29.926628 22012 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:29.905793 21247 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:29.928192 21274 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 02:59:29.928992 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 99756.4us
I20251214 02:59:29.931921 23296 raft_consensus.cc:993] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:29.931078 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.941248 21201 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:29.941430 21029 catalog_manager.cc:5654] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 3 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:29.946864 21986 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.942540 21202 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:29.944964 22016 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:29.930279 21276 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:29.949119 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:29.951990 23257 raft_consensus.cc:1081] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:29.968200 21099 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:29.974664 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:29.959429 23242 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:29.970690 23199 raft_consensus.cc:2764] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251214 02:59:29.987630 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:29.988552 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:29.992486 23209 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20251214 02:59:29.993153 23228 raft_consensus.cc:2764] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251214 02:59:29.994050 23237 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election won for term 3
W20251214 02:59:29.995220 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 89526.6us
I20251214 02:59:29.996831 21098 leader_election.cc:400] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:29.978973 23186 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:30.009452 23209 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:30.009603 23247 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:30.011519 21098 leader_election.cc:403] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:30.022346 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.023105 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:30.023625 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:30.028786 23173 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4.
I20251214 02:59:30.031813 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:30.012310 23186 raft_consensus.cc:697] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:30.012843 23220 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:30.039981 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.040655 21281 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:30.013321 23209 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:30.059944 21278 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:30.059820 21173 leader_election.cc:400] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:30.061733 23257 consensus_queue.cc:1059] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:30.062661 23257 raft_consensus.cc:3055] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:30.062973 23257 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.063803 23257 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.039408 23220 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:30.066248 23220 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:30.068651 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:30.069237 23209 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.070482 23209 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:30.070986 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:30.071550 21273 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:30.025151 23247 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.019135 23237 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:30.077077 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:30.075919 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.013231 23129 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20251214 02:59:30.078419 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 81779.8us
I20251214 02:59:30.082764 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:30.083377 21129 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:30.079784 21133 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:30.086123 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:30.090930 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.099406 23214 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:30.099979 23214 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.100850 23214 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.084363 23257 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:30.107182 21278 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:30.090597 23295 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:30.116639 21281 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 02:59:30.147439 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56601.9us
I20251214 02:59:30.094419 23112 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20251214 02:59:30.148238 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.150024 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.152280 21133 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:30.151538 21985 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:30.158635 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 126845us
I20251214 02:59:30.160140 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.147488 21129 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:30.161274 21129 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:30.161414 23295 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:30.161695 23324 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20251214 02:59:30.164469 21129 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:30.166218 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:30.166601 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 57894.1us
I20251214 02:59:30.152153 23186 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.155777 23187 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:30.158268 23322 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:30.158126 23193 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:30.177978 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:30.191802 22016 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20251214 02:59:30.187220 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 101251us
I20251214 02:59:30.198401 21027 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:30.205447 21131 raft_consensus.cc:2410] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251214 02:59:30.213724 21199 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:30.215767 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:30.217633 22018 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:30.211899 23193 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.267066 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:30.217555 21129 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.272761 21098 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:30.275146 21276 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 5
I20251214 02:59:30.276573 21278 raft_consensus.cc:1275] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:30.210454 23187 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.184607 23173 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:30.269457 21027 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:30.279533 23187 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:30.281325 21276 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:30.280956 21027 catalog_manager.cc:5654] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:30.194731 23237 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.283150 23288 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:30.283819 23217 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:30.285370 22016 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:30.216738 23322 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:30.182169 23263 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:30.286530 23322 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.286687 23263 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.288858 23263 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:30.275146 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.291170 21277 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:30.292358 21129 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:30.302690 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 133309us
I20251214 02:59:30.304145 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.307029 23340 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:30.307494 23340 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:30.307823 23340 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:30.321048 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.323069 23221 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:30.323509 23221 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:30.323843 23221 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:30.336126 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.337200 23112 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:30.337707 23112 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.339040 23112 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.212541 23247 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:30.363313 21279 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:30.367544 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.399557 21274 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:30.364714 21129 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:30.362752 23129 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:30.404217 23263 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20251214 02:59:30.397130 23217 consensus_queue.cc:1048] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20251214 02:59:30.407677 23288 consensus_queue.cc:1059] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:30.414317 22020 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 02:59:30.345930 23340 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.417976 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:30.422189 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:30.432065 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:30.433300 23187 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:30.301674 23199 raft_consensus.cc:2764] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20251214 02:59:30.440768 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 277484us
I20251214 02:59:30.446985 21247 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.456984 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:30.353662 23257 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20251214 02:59:30.444330 23263 raft_consensus.cc:3055] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:30.354034 23173 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20251214 02:59:30.378154 23221 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.408586 23220 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.484078 21200 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:30.464366 23186 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:30.409629 23296 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20251214 02:59:30.461194 21274 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:30.481217 22020 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:30.481024 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42016
I20251214 02:59:30.486819 21275 raft_consensus.cc:1240] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:30.487912 21199 raft_consensus.cc:606] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Received request to transfer leadership
I20251214 02:59:30.486857 21098 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:30.490880 23217 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:30.456822 23164 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20251214 02:59:30.409129 23193 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.465608 23263 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:30.485167 23205 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20251214 02:59:30.492712 23221 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:30.503365 21277 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:30.490779 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:30.508518 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.513561 21206 raft_consensus.cc:3055] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:30.513949 21206 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.515658 23295 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:30.516314 21206 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.517345 21206 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:30.519294 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 107621us
W20251214 02:59:30.519973 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 65794.5us
I20251214 02:59:30.525087 21203 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:30.526542 23186 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:30.527653 23288 raft_consensus.cc:3055] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:30.528017 23288 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:30.528820 23288 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.529845 23288 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:30.554853 21133 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:30.519431 23274 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:30.555327 23247 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:30.573627 21277 raft_consensus.cc:1275] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:30.559321 23220 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:30.579172 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:30.580514 21125 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:30.579332 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:30.581244 21129 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:30.579416 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:30.582063 21127 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:30.589190 21206 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:30.590456 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.591049 21274 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 6
W20251214 02:59:30.596542 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 88093.4us
I20251214 02:59:30.575681 23274 raft_consensus.cc:697] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:30.598237 23340 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:30.599136 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.598917 23274 consensus_queue.cc:237] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.599694 21273 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 7
I20251214 02:59:30.599843 21274 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:30.603488 23247 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:30.604040 23247 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.569368 23228 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20251214 02:59:30.575209 23112 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20251214 02:59:30.620676 21127 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:30.633800 21125 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 02:59:30.637758 23348 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.083s user 0.002s sys 0.000s
W20251214 02:59:30.638899 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:30.639474 21130 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:30.642715 21129 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:30.565109 23263 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.651886 21247 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.653005 23164 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:30.653398 23247 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:30.660290 23356 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:30.662698 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:30.649343 21203 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:30.671169 21201 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:30.648370 23358 log.cc:927] Time spent T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.115s user 0.000s sys 0.005s
I20251214 02:59:30.689670 21131 raft_consensus.cc:1240] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:30.690244 23242 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20251214 02:59:30.702806 23173 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:30.720292 21273 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:30.648660 23213 consensus_queue.cc:1048] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20251214 02:59:30.700526 23263 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:30.746719 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:30.747459 23278 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:30.748967 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.749449 23205 raft_consensus.cc:993] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:30.750327 22016 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:30.769001 21206 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42016
I20251214 02:59:30.783200 23247 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20251214 02:59:30.767057 23278 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.907825 23224 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:30.782650 23187 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:30.908262 23224 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:30.908919 23187 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.909449 23278 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:30.909338 23224 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.909824 23187 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.910951 23224 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:30.773887 23112 raft_consensus.cc:1081] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:30.912880 21124 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20251214 02:59:30.913054 21202 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:30.914896 21124 pending_rounds.cc:85] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 4
I20251214 02:59:30.916339 21124 pending_rounds.cc:107] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20251214 02:59:30.917289 21124 raft_consensus.cc:2889] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:30.918259 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:30.919770 23278 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:30.920568 21199 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251214 02:59:30.907732 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:30.923633 21278 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:30.926185 23278 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:30.937983 23278 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:30.930497 21124 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:30.932072 23199 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:30.945046 21131 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20251214 02:59:30.945425 21131 pending_rounds.cc:85] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 3
I20251214 02:59:30.945654 21131 pending_rounds.cc:107] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251214 02:59:30.945888 21131 raft_consensus.cc:2889] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:30.934204 21130 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:30.926822 23112 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:30.941010 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 280) took 3133 ms. Trace:
I20251214 02:59:30.943938 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:30.944443 21127 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:30.954946 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:30.955312 21133 raft_consensus.cc:2368] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:30.956858 21027 rpcz_store.cc:276] 1214 02:59:27.808042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:27.808427 (+ 385us) service_pool.cc:225] Handling call
1214 02:59:30.940946 (+3132519us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:30.457010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:30.563019 (+106009us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:30.565928 (+ 2909us) write_op.cc:435] Acquired schema lock
1214 02:59:30.568650 (+ 2722us) tablet.cc:660] Decoding operations
1214 02:59:30.646427 (+ 77777us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:30.649991 (+ 3564us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:30.650020 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
1214 02:59:30.650334 (+ 314us) tablet.cc:699] Row locks acquired
1214 02:59:30.650352 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:30.650489 (+ 137us) write_op.cc:270] Start()
1214 02:59:30.653611 (+ 3122us) write_op.cc:276] Timestamp: P: 1765681170650444 usec, L: 0
1214 02:59:30.653636 (+ 25us) op_driver.cc:348] REPLICATION: starting
1214 02:59:30.654210 (+ 574us) log.cc:844] Serialized 1375 byte log entry
1214 02:59:30.656360 (+ 2150us) op_driver.cc:464] REPLICATION: finished
1214 02:59:30.656790 (+ 430us) write_op.cc:301] APPLY: starting
1214 02:59:30.656872 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:30.656998 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:30.657107 (+ 109us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:30.657590 (+ 483us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:30.657761 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1214 02:59:30.657806 (+ 45us) write_op.cc:312] APPLY: finished
1214 02:59:30.658011 (+ 205us) log.cc:844] Serialized 35 byte log entry
1214 02:59:30.658172 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:30.658270 (+ 98us) write_op.cc:454] Released schema lock
1214 02:59:30.658313 (+ 43us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:30.659911 (+ 1598us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:30.659951 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":125,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":1802,"apply.run_wall_time_us":1801,"num_ops":3,"prepare.queue_time_us":307,"prepare.run_cpu_time_us":3636,"prepare.run_wall_time_us":208031,"replication_time_us":2571}]]}
I20251214 02:59:30.959914 23288 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:30.960460 23112 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:30.961570 23357 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:30.961474 23112 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.965454 23351 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:30.962116 23357 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:30.971312 23357 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:30.973217 23363 log.cc:927] Time spent T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.284s user 0.001s sys 0.000s
W20251214 02:59:30.973346 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 63274.7us
I20251214 02:59:30.976075 21247 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:30.977878 23164 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:30.980036 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:30.980688 21277 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 9
I20251214 02:59:30.989048 21277 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:30.990387 23164 consensus_queue.cc:1059] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:31.038762 23164 raft_consensus.cc:3055] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:31.039255 23164 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:31.040241 23164 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.041277 23228 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:31.042382 21200 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 7
W20251214 02:59:31.042951 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 110057us
I20251214 02:59:31.043999 23173 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47052 (request call id 922) took 1127 ms. Trace:
I20251214 02:59:31.044459 23173 rpcz_store.cc:276] 1214 02:59:29.916139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:29.916290 (+ 151us) service_pool.cc:225] Handling call
1214 02:59:29.921391 (+ 5101us) log.cc:844] Serialized 501 byte log entry
1214 02:59:30.162698 (+241307us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b1800255e20. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6714d41f21 00007f6714d4e901 00007f670be6533a 00007f67154ea5c9 00007f670bd25030 00007f670beed0d4 00007f670beed045 00007f670eef7b3b 00007f670ef7c893 00007f670efa43d2 00007f670ef9c67a
1214 02:59:30.617867 (+455169us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b5400375620. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef13dee 00007f670ef1e960 00007f670ef1e908 00007f670ef1e888 00007f670ef1e843 00007f670ef1dacc 00007f6714dd1cc6 00007f6714dcc478 00007f670ef5a4fb 00007f670ef5cff0
1214 02:59:31.043916 (+426049us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35669,"raft.queue_time_us":88908,"raft.run_cpu_time_us":1352,"raft.run_wall_time_us":89010,"spinlock_wait_cycles":25521152,"thread_start_us":155564,"threads_started":1,"wal-append.queue_time_us":277704}
I20251214 02:59:31.047499 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 6ae31f8b580840eb863e4d400367c1be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:31.048950 23164 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:31.049340 21125 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:30.996356 23199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:50512 (request call id 341) took 4239 ms. Trace:
I20251214 02:59:31.050448 23199 rpcz_store.cc:276] 1214 02:59:26.757343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:26.757493 (+ 150us) service_pool.cc:225] Handling call
1214 02:59:26.761565 (+ 4072us) log.cc:844] Serialized 501 byte log entry
1214 02:59:30.996246 (+4234681us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161854,"raft.queue_time_us":162214,"raft.run_cpu_time_us":732,"raft.run_wall_time_us":50275,"thread_start_us":458149,"threads_started":1,"wal-append.queue_time_us":415659,"wal-append.run_cpu_time_us":1590,"wal-append.run_wall_time_us":1040193}
I20251214 02:59:31.052448 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet d55cde2542d544c5b99e6113b95910b8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20251214 02:59:31.012174 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 53095.9us
I20251214 02:59:31.015908 23205 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 4 to 7, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:31.020025 22012 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 4 to 7, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:31.022931 21274 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:31.025352 21275 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:31.024991 21030 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:31.027729 21273 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50516
I20251214 02:59:31.033591 23242 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:31.066141 23242 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.068315 23242 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.069144 21273 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:31.069610 21273 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:31.073796 21200 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:30.991935 21247 leader_election.cc:400] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:31.087615 21247 leader_election.cc:403] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20251214 02:59:31.088580 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 96781.6us
I20251214 02:59:31.089221 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.089816 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:31.075815 21030 catalog_manager.cc:5654] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 2 to 3, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:31.049309 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:31.092878 21273 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:31.093199 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 43975.2us
I20251214 02:59:31.081619 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.106976 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:31.106849 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:31.111436 21273 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:31.103293 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:31.120935 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:31.122795 21124 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:31.096400 23278 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:31.094461 23224 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:31.133471 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:31.136839 21173 leader_election.cc:400] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:31.137224 21173 leader_election.cc:403] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:31.160038 21201 raft_consensus.cc:1275] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:31.138307 23274 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.136292 23295 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4.
I20251214 02:59:31.156134 23214 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:31.132212 23213 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20251214 02:59:31.127140 23112 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20251214 02:59:31.126487 23296 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20251214 02:59:31.164113 23391 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:31.185281 23217 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.157956 23240 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20251214 02:59:31.261981 21207 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 02:59:31.182668 23292 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251214 02:59:31.270237 22014 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20251214 02:59:31.193307 23349 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20251214 02:59:31.219198 23274 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.165906 22016 raft_consensus.cc:2955] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:31.341461 23371 log.cc:927] Time spent T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20251214 02:59:31.342116 23295 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:31.342727 23295 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.257421 23217 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.344774 23295 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:31.345315 23217 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.346305 23186 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20251214 02:59:31.385525 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 224965us
W20251214 02:59:31.393013 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 254830us
W20251214 02:59:31.394865 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 284940us
I20251214 02:59:31.397820 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:31.386304 23390 log.cc:927] Time spent T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251214 02:59:31.382297 23217 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.408195 23217 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.410705 23324 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:31.411193 23324 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.403918 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.413002 23324 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.414368 23221 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.414858 23221 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.416115 21280 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:31.418255 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:31.418879 22014 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:31.419135 21281 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:31.324023 23224 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.070s
I20251214 02:59:31.402662 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.405220 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.406349 21276 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:31.424841 21207 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:31.443950 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:31.418256 23221 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.423455 23274 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.461167 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.475425 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:31.460784 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 53122.8us
I20251214 02:59:31.406394 23393 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:31.481371 23393 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.483506 23322 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:31.484020 23322 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.484496 23201 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:31.503898 21099 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:31.506675 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.506583 23221 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:31.461277 23217 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:31.510062 23217 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:31.510432 23217 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:31.511328 23217 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:31.516104 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.522357 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.523383 21201 raft_consensus.cc:2393] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:31.460688 23278 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:31.541831 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:31.544375 21203 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:31.545975 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.547842 23224 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20251214 02:59:31.552181 23392 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:31.555248 23393 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:31.558095 23407 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:31.558449 23247 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:31.558651 23407 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.560703 23407 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:31.561321 23392 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.562021 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.562357 23395 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:31.561794 23392 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.564240 23392 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:31.564718 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.559592 23217 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.565210 23322 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.549289 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.569380 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.569995 21281 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:31.561941 23407 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:31.542986 23164 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:31.573830 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.574182 23240 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:31.575117 23217 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.576020 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:31.576174 22014 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:31.576598 21207 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:31.574553 21273 raft_consensus.cc:2393] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:31.606205 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.619586 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:31.629377 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.630378 21203 raft_consensus.cc:2410] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251214 02:59:31.633050 23196 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:31.642776 23360 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20251214 02:59:31.643927 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.589215 23407 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.645675 23349 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:31.652096 21206 raft_consensus.cc:606] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Received request to transfer leadership
I20251214 02:59:31.652755 23407 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.652887 21280 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:31.653847 21282 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:31.654469 21098 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:31.655406 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:31.646061 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.642674 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:31.645835 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:31.649988 23408 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:31.659562 23408 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.660145 23407 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:31.650159 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:31.670653 21128 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:31.656428 21274 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:31.641077 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:31.672048 21273 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 5
I20251214 02:59:31.669226 23173 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:31.658947 23217 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:31.676784 21276 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:31.679136 21207 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
W20251214 02:59:31.689538 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 173668us
I20251214 02:59:31.650367 23278 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.701370 23221 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:31.708340 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.709801 21273 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:31.643895 23403 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20251214 02:59:31.623404 23187 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20251214 02:59:31.711850 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:31.709097 21203 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:31.685966 23173 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:31.661492 23408 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.779220 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:31.787721 23400 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20251214 02:59:31.789425 23413 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:31.789989 23413 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:31.727161 23357 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20251214 02:59:31.791339 22016 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:31.793701 23413 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:31.795137 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.777175 23173 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:31.800643 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:31.801421 22014 raft_consensus.cc:2410] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251214 02:59:31.803769 21245 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.805464 21986 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:31.805653 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.808446 21986 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:31.809350 21245 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.813055 21124 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20251214 02:59:31.813459 21124 pending_rounds.cc:85] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 3
I20251214 02:59:31.817553 21124 pending_rounds.cc:107] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251214 02:59:31.817834 21124 raft_consensus.cc:2889] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:31.819284 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:31.814205 21201 raft_consensus.cc:2955] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:31.809392 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 139942us
I20251214 02:59:31.836323 21098 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:31.838753 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.843537 21128 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:31.844795 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:31.817152 23324 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:31.847277 21098 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:31.813630 23164 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:31.817492 23171 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:31.813123 23414 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:31.817757 23224 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20251214 02:59:31.851202 23324 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:31.871980 23274 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:31.896843 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.895704 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:31.882380 23217 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:31.881981 23421 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:31.922304 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 106069us
I20251214 02:59:31.891078 23324 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:31.947314 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:31.923080 23274 raft_consensus.cc:697] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:31.965160 23405 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:31.977345 23371 log.cc:927] Time spent T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20251214 02:59:32.078276 23187 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:32.079001 23187 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.081220 23187 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:32.131649 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 236831us
W20251214 02:59:32.147527 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 136222us
W20251214 02:59:32.147070 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 137034us
I20251214 02:59:32.163807 23274 consensus_queue.cc:237] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:31.978824 23421 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.285883 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:32.417824 21276 raft_consensus.cc:1275] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251214 02:59:32.419188 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 471962us
I20251214 02:59:32.420011 21247 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:32.380362 23414 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:32.423816 23414 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.426365 23414 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.430207 21126 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:32.008177 23406 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:32.437727 21126 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20251214 02:59:32.417017 23424 log.cc:927] Time spent T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.514s user 0.001s sys 0.000s
W20251214 02:59:32.435861 21125 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.007s sys 0.023s
W20251214 02:59:32.450131 21125 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s user 0.008s sys 0.023s
I20251214 02:59:32.454733 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:32.455451 21276 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:32.326706 23221 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:32.464049 21130 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:32.318980 23112 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:32.447708 23398 log.cc:927] Time spent T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251214 02:59:32.445417 23396 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:32.170336 23173 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:32.474267 21201 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.012s sys 0.038s
W20251214 02:59:32.475059 21201 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.510s user 0.013s sys 0.038s
W20251214 02:59:32.476598 23416 log.cc:927] Time spent T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.554s user 0.000s sys 0.001s
I20251214 02:59:32.469216 23112 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.488891 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:32.474244 23408 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:32.477061 23396 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:32.522800 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:32.583645 21130 raft_consensus.cc:2368] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 02:59:32.484759 23413 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:32.466361 23221 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:32.641022 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 208906us
I20251214 02:59:32.484463 23292 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.147s
I20251214 02:59:32.582245 23417 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:32.632941 23415 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.624s user 0.001s sys 0.000s
W20251214 02:59:32.672673 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 43481.7us
I20251214 02:59:32.663913 21131 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 02:59:32.675596 22018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44274 (request call id 474) took 1180 ms. Trace:
I20251214 02:59:32.630481 23399 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:32.676156 22018 rpcz_store.cc:276] 1214 02:59:31.488782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.488883 (+ 101us) service_pool.cc:225] Handling call
1214 02:59:31.511026 (+ 22143us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.530794 (+ 19768us) raft_consensus.cc:1534] Early marking committed up to index 3
1214 02:59:31.530824 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.541295 (+ 10471us) log.cc:844] Serialized 26 byte log entry
1214 02:59:31.800403 (+259108us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:31.800452 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:31.800518 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.668947 (+868429us) raft_consensus.cc:1685] finished
1214 02:59:32.668979 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.669179 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19354,"raft.queue_time_us":36270,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":19877,"thread_start_us":210756,"threads_started":1,"wal-append.queue_time_us":289834}
I20251214 02:59:32.691856 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:32.626927 23408 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.695668 23408 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:32.700104 23442 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251214 02:59:32.700990 23221 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:32.662602 23224 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:32.702024 23224 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.704077 23224 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:32.717851 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 83765.3us
I20251214 02:59:32.655128 23413 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.725327 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:32.708453 23407 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.018s sys 0.001s
I20251214 02:59:32.733263 23413 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.736088 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:32.722985 23391 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:32.747080 23438 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:32.748649 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1200) took 1220 ms. Trace:
W20251214 02:59:32.748863 23418 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.384s user 0.001s sys 0.000s
W20251214 02:59:32.731393 23407 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s user 0.019s sys 0.001s
I20251214 02:59:32.742797 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:32.743400 23445 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:32.724814 23396 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:32.744160 23428 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
I20251214 02:59:32.727303 23421 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.768807 21201 rpcz_store.cc:276] 1214 02:59:31.527525 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.528672 (+ 1147us) service_pool.cc:225] Handling call
1214 02:59:31.548070 (+ 19398us) raft_consensus.cc:1471] Updating replica for 2 ops
1214 02:59:31.555976 (+ 7906us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.556004 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1214 02:59:31.556899 (+ 895us) log.cc:844] Serialized 527 byte log entry
1214 02:59:31.814088 (+257189us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:31.924539 (+110451us) log.cc:844] Serialized 14 byte log entry
1214 02:59:32.475612 (+551073us) log.cc:844] Serialized 14 byte log entry
1214 02:59:32.476803 (+ 1191us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.476870 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.747925 (+271055us) raft_consensus.cc:1685] finished
1214 02:59:32.747943 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.748060 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":724565,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":11393,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":3210,"spinlock_wait_cycles":9570176,"thread_start_us":767706,"threads_started":2,"wal-append.queue_time_us":364669}
I20251214 02:59:32.777923 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:32.785328 21202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1203) took 1218 ms. Trace:
W20251214 02:59:32.796067 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 70841.9us
I20251214 02:59:32.797853 21202 rpcz_store.cc:276] 1214 02:59:31.567054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.567185 (+ 131us) service_pool.cc:225] Handling call
1214 02:59:31.567462 (+ 277us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.567765 (+ 303us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.567790 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.577044 (+ 9254us) log.cc:844] Serialized 26 byte log entry
1214 02:59:31.839537 (+262493us) raft_consensus.cc:1645] Marking committed up to 7
1214 02:59:31.839573 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:31.839632 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.785094 (+945462us) raft_consensus.cc:1685] finished
1214 02:59:32.785120 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.785264 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18621,"raft.queue_time_us":390,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":51058,"thread_start_us":194651,"threads_started":1,"wal-append.queue_time_us":344951}
I20251214 02:59:32.799535 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 257) took 1270 ms. Trace:
I20251214 02:59:32.799954 21278 rpcz_store.cc:276] 1214 02:59:31.529613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:32.798842 (+1269229us) service_pool.cc:225] Handling call
1214 02:59:32.799109 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:32.799255 (+ 146us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:32.799276 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:32.799309 (+ 33us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:32.799335 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.799396 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.799486 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:32.724665 23357 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20251214 02:59:32.804944 21280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1762) took 1153 ms. Trace:
I20251214 02:59:32.681005 23417 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.811909 23417 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:32.689131 23399 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.814275 23399 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.746421 23391 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.817198 23391 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.818003 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:32.747602 23438 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:32.820475 23438 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:32.738847 23449 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:32.743338 23435 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:32.748941 23360 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:32.824811 22014 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:32.807523 21280 rpcz_store.cc:276] 1214 02:59:31.651529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.653541 (+ 2012us) service_pool.cc:225] Handling call
1214 02:59:32.318519 (+664978us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b5800235598. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:32.318719 (+ 200us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:32.327420 (+ 8701us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:32.327457 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:32.353769 (+ 26312us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.668194 (+314425us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:32.668242 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.668326 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.804694 (+136368us) raft_consensus.cc:1685] finished
1214 02:59:32.804712 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.804888 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8295,"raft.queue_time_us":10227,"spinlock_wait_cycles":71522944,"thread_start_us":296857,"threads_started":1,"wal-append.queue_time_us":367380}
I20251214 02:59:32.786036 23445 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.893090 21124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1763) took 1010 ms. Trace:
W20251214 02:59:32.926144 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 123637us
W20251214 02:59:32.977568 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 1.1283e+06us
I20251214 02:59:32.870533 21245 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:32.921794 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1759) took 1075 ms. Trace:
I20251214 02:59:32.911679 21277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1764) took 1259 ms. Trace:
I20251214 02:59:32.826073 23391 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:32.982110 21099 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:32.853776 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 179712us
W20251214 02:59:32.982230 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 111816us
I20251214 02:59:32.858521 23430 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:32.807092 23173 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:32.854519 21133 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.035s sys 0.031s
I20251214 02:59:32.989816 21099 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:32.986361 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:32.998234 21133 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.720s user 0.052s sys 0.069s
I20251214 02:59:32.988655 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:32.999214 21278 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:32.982417 21277 rpcz_store.cc:276] 1214 02:59:31.651992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.670624 (+ 18632us) service_pool.cc:225] Handling call
1214 02:59:31.691219 (+ 20595us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.713874 (+ 22655us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.713907 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.740134 (+ 26227us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.417979 (+677845us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:32.418020 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.418086 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.911440 (+493354us) raft_consensus.cc:1685] finished
1214 02:59:32.911463 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.911590 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31409,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":38527,"thread_start_us":104446,"threads_started":1,"wal-append.queue_time_us":547106}
I20251214 02:59:32.982252 21128 rpcz_store.cc:276] 1214 02:59:31.846810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.848403 (+ 1593us) service_pool.cc:225] Handling call
1214 02:59:31.848762 (+ 359us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:32.431500 (+582738us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:32.431529 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:32.450361 (+ 18832us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.733920 (+283559us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:32.733961 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.734026 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.921535 (+187509us) raft_consensus.cc:1685] finished
1214 02:59:32.921558 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.921723 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79565,"raft.queue_time_us":122035,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13433,"thread_start_us":746992,"threads_started":2,"wal-append.queue_time_us":392536}
W20251214 02:59:32.904330 23429 log.cc:927] Time spent T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.273s user 0.001s sys 0.000s
I20251214 02:59:33.008829 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:32.875723 23443 log.cc:927] Time spent T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20251214 02:59:33.011349 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 1130) took 1385 ms. Trace:
I20251214 02:59:33.020028 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:32.859755 23441 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:32.981101 21124 rpcz_store.cc:276] 1214 02:59:31.882731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.882912 (+ 181us) service_pool.cc:225] Handling call
1214 02:59:31.890160 (+ 7248us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:32.447380 (+557220us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:32.447414 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:32.459699 (+ 12285us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.658660 (+198961us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:32.658692 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.658755 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:32.892897 (+234142us) raft_consensus.cc:1685] finished
1214 02:59:32.892916 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:32.893033 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53400,"raft.queue_time_us":95287,"spinlock_wait_cycles":4784896,"thread_start_us":661434,"threads_started":2,"wal-append.queue_time_us":261382}
W20251214 02:59:32.947054 23324 consensus_meta.cc:287] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.975s user 0.010s sys 0.001s
I20251214 02:59:32.890183 23446 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:33.020689 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.012488 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:32.876226 23360 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:32.855818 23112 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.024591 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:32.875571 23440 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:32.920101 23465 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:32.985081 23392 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:33.029526 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:32.989114 23444 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:33.041847 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1760) took 1190 ms. Trace:
I20251214 02:59:33.035917 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.048446 21132 rpcz_store.cc:276] 1214 02:59:31.851814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.852120 (+ 306us) service_pool.cc:225] Handling call
1214 02:59:31.871443 (+ 19323us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:32.453406 (+581963us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:32.453434 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:32.453470 (+ 36us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:32.453499 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:33.041637 (+588138us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540002fdd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:33.041680 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:33.041794 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1338122112}
I20251214 02:59:33.019145 21133 rpcz_store.cc:276] 1214 02:59:31.625986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.626095 (+ 109us) service_pool.cc:225] Handling call
1214 02:59:31.629448 (+ 3353us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.805227 (+175779us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.805255 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.814929 (+ 9674us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.009339 (+194410us) raft_consensus.cc:1645] Marking committed up to 7
1214 02:59:32.998783 (+989444us) log.cc:844] Serialized 14 byte log entry
1214 02:59:33.011054 (+ 12271us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:33.011121 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:33.011160 (+ 39us) raft_consensus.cc:1685] finished
1214 02:59:33.011171 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:33.011280 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35653,"prepare.queue_time_us":324823,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":44476,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":103,"thread_start_us":1005930,"threads_started":3,"wal-append.queue_time_us":194953}
I20251214 02:59:32.898937 23340 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:33.019879 23421 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:33.049450 21126 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:33.056706 21173 leader_election.cc:400] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 02:59:33.057040 21173 leader_election.cc:403] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:33.052417 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.058195 21279 raft_consensus.cc:2393] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:33.058521 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.059171 21201 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:33.031097 23173 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:33.032739 23296 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20251214 02:59:33.053913 23360 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.057868 23340 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.063136 23360 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.063881 23340 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.093453 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:33.058622 21279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50516 (request call id 1759) took 1407 ms (client timeout 1685 ms). Trace:
I20251214 02:59:33.095080 21125 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1762) took 1236 ms. Trace:
I20251214 02:59:33.095818 21278 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
W20251214 02:59:33.095235 21279 rpcz_store.cc:269] 1214 02:59:31.650723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.654344 (+ 3621us) service_pool.cc:225] Handling call
1214 02:59:33.052275 (+1397931us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b5800235598. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f53bd0 00007f6713f534b4 00007f670d4e3834 00007f670d4e37c2
1214 02:59:33.058537 (+ 6262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15056768}
I20251214 02:59:33.095492 21125 rpcz_store.cc:276] 1214 02:59:31.858234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.858356 (+ 122us) service_pool.cc:225] Handling call
1214 02:59:31.858738 (+ 382us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.870725 (+ 11987us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.870758 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.871252 (+ 494us) log.cc:844] Serialized 26 byte log entry
1214 02:59:32.450482 (+579230us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:32.450522 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:32.450611 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:33.094894 (+644283us) raft_consensus.cc:1685] finished
1214 02:59:33.094912 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:33.095027 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49079,"raft.queue_time_us":28629,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":21274,"thread_start_us":578949,"threads_started":1,"wal-append.queue_time_us":605204}
I20251214 02:59:32.859484 23435 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.054451 23440 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.104210 23435 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.104787 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:33.105417 21205 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 6
I20251214 02:59:33.037804 23441 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.107460 23440 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.058451 23278 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:33.109304 23278 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:32.850327 23449 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.005259 23391 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:33.115691 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 59140.5us
I20251214 02:59:33.121344 21273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1768) took 1323 ms. Trace:
I20251214 02:59:33.127085 21205 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:33.127485 21273 rpcz_store.cc:276] 1214 02:59:31.797630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.798306 (+ 676us) service_pool.cc:225] Handling call
1214 02:59:31.798621 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.807288 (+ 8667us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.807316 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:31.807750 (+ 434us) log.cc:844] Serialized 26 byte log entry
1214 02:59:31.896204 (+ 88454us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:31.896241 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:31.896300 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:33.121144 (+1224844us) raft_consensus.cc:1685] finished
1214 02:59:33.121168 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:33.121285 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2296,"raft.queue_time_us":8503,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":88114,"threads_started":1,"wal-append.queue_time_us":658975}
I20251214 02:59:33.031317 23399 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.067s
I20251214 02:59:33.095672 23465 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.004786 23430 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:33.132686 23411 log.cc:927] Time spent T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 1.426s user 0.007s sys 0.004s
I20251214 02:59:33.133242 23278 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.037211 23324 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.141050 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.144022 21099 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:33.137814 23400 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20251214 02:59:33.151888 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:33.146932 23449 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.148424 23445 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.040503 23463 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.131s
I20251214 02:59:33.131716 23193 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 02:59:33.149993 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.131281 23465 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.153060 23242 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:33.163995 23242 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:33.153082 23402 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:33.061204 23173 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.174955 21099 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:33.164858 21200 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:33.193343 21200 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:33.198596 23408 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:33.199267 23408 raft_consensus.cc:697] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:33.200202 23408 consensus_queue.cc:237] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.203145 23217 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.203737 23217 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.209049 23217 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.209328 23421 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.229622 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.194275 21200 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.164388 23242 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
W20251214 02:59:33.232282 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57865us
I20251214 02:59:33.232455 21200 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:33.151947 23324 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.051175 23446 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.210392 23199 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:33.288895 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 53521.5us
I20251214 02:59:33.209939 23421 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.234622 23431 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:33.292680 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 266240us
I20251214 02:59:33.290762 22012 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:33.291183 21277 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
W20251214 02:59:33.285202 23409 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.572s user 0.016s sys 0.007s
I20251214 02:59:33.295794 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.295774 21098 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:33.279698 23440 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.297211 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:33.280543 23402 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.279911 23460 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 02:59:33.300233 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 292556us
I20251214 02:59:33.280319 23430 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.310094 21200 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:33.307418 21282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1763) took 1655 ms. Trace:
I20251214 02:59:33.312847 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:33.297768 23409 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.590s user 0.016s sys 0.007s
I20251214 02:59:33.323323 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.300654 23263 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.299229 23431 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.331343 23391 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.305873 23392 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.327219 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.332866 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.333668 21276 raft_consensus.cc:2410] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251214 02:59:33.057945 22016 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:33.334429 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:33.335136 21276 raft_consensus.cc:2393] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:33.335808 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.336725 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.327764 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:33.328738 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:33.317030 23205 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20251214 02:59:33.330533 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.332331 23431 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.302011 23435 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:33.342839 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.324108 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.341302 23356 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:33.348889 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.338080 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.350716 23466 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.338441 21985 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:33.344154 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.290805 23199 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.351894 22018 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:33.386247 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.344997 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.345175 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.317818 21282 rpcz_store.cc:276] 1214 02:59:31.651765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.654507 (+ 2742us) service_pool.cc:225] Handling call
1214 02:59:31.662865 (+ 8358us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:31.669885 (+ 7020us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:31.669914 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:32.922520 (+1252606us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b2c00107c08. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670d89c9b1 00007f670d8abfe2 00007f670efd9851 00007f670ef9e0ba 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22
1214 02:59:32.923756 (+ 1236us) log.cc:844] Serialized 26 byte log entry
1214 02:59:33.197453 (+273697us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:33.197489 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:33.198212 (+ 723us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:33.287427 (+ 89215us) raft_consensus.cc:1685] finished
1214 02:59:33.287445 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:33.307354 (+ 19909us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9570,"raft.queue_time_us":6794,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8491,"spinlock_wait_cycles":18470784,"thread_start_us":219428,"threads_started":1,"wal-append.queue_time_us":253784}
I20251214 02:59:33.346196 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.344272 23263 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.349905 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.325866 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.319136 23465 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:33.333591 23173 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.378712 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.313324 23405 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.310828 23393 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:33.383556 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45054.4us
W20251214 02:59:33.391511 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 53913.4us
I20251214 02:59:33.351274 23466 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.402361 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.405807 21273 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 02:59:33.332799 23392 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.308789 23440 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.414556 22019 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:33.414294 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.424834 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.399605 23445 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:33.425933 21279 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:33.435087 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.468950 23461 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:33.402331 23112 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:33.415498 23199 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.306115 23460 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.316473 23402 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:33.787901 23472 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.438s user 0.001s sys 0.000s
W20251214 02:59:33.787241 21248 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.66:44049 timed out after 2.396s (SENT) [suppressed 26 similar messages]
I20251214 02:59:33.749039 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:33.786048 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.796882 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.797243 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:33.751459 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.749495 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:33.513829 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 117714us
I20251214 02:59:33.799203 21207 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:33.787521 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:33.535799 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 137747us
I20251214 02:59:33.751554 21096 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:33.793867 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 44948.1us
I20251214 02:59:33.484094 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:33.513815 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43744.4us
I20251214 02:59:33.484913 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.758069 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.657642 23421 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.799818 23112 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.657847 23242 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.815316 21125 raft_consensus.cc:2368] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9.
I20251214 02:59:33.818171 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:33.823674 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.824366 21282 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
W20251214 02:59:33.747839 23481 log.cc:927] Time spent T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
I20251214 02:59:33.828193 21171 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:33.534600 23399 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:33.513988 23221 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.517620 23422 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.574319 23447 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.661885 23446 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.689831 23480 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.482915 23470 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:33.853920 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.856178 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:33.411566 23393 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.546336 23412 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.797753 23224 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.517360 23474 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:33.526039 23444 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.862120 21986 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:33.438840 23445 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.815394 23421 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.842339 23263 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.841761 23247 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.841732 23459 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.876003 23296 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:33.858451 23221 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.895495 23445 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:33.897294 21986 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:33.886523 23447 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.419698 23405 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.920500 23447 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.920814 23405 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.876538 23392 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.921097 21129 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:33.877779 23422 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.922236 23356 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:33.896697 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:33.857424 23399 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:33.928499 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:33.890950 23440 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.899210 23465 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:33.896107 23444 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.891542 23470 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.902899 23242 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.903113 23296 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:33.899861 23466 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.955365 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:33.903734 23263 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.961550 23392 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.902097 23460 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.892251 23224 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.886719 23480 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.900854 23421 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.947388 23112 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.968952 23394 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:33.897433 23492 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.841485 23461 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:33.958354 23435 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.964179 23296 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:33.967769 23393 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:33.976810 23399 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.967769 23470 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:33.982304 21202 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:33.916714 23247 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.027544 23247 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:33.972204 23490 log.cc:927] Time spent T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20251214 02:59:33.943022 23459 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.029852 23459 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:33.989840 23487 log.cc:927] Time spent T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
W20251214 02:59:33.999440 21200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42032 (request call id 1192) took 2892 ms (client timeout 2396 ms). Trace:
W20251214 02:59:34.003177 23488 log.cc:927] Time spent T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.357s user 0.000s sys 0.001s
I20251214 02:59:33.964576 23405 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:34.031390 21200 rpcz_store.cc:269] 1214 02:59:31.106549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.106760 (+ 211us) service_pool.cc:225] Handling call
1214 02:59:33.999233 (+2892473us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54001464d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:33.999346 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4722097536}
I20251214 02:59:34.031786 23405 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.033706 23405 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.019863 22016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44274 (request call id 481) took 1041 ms. Trace:
W20251214 02:59:34.021031 23129 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.010s sys 0.011s
I20251214 02:59:34.034312 22016 rpcz_store.cc:276] 1214 02:59:32.978824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:32.980850 (+ 2026us) service_pool.cc:225] Handling call
1214 02:59:33.019279 (+ 38429us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.052315 (+ 33036us) raft_consensus.cc:1534] Early marking committed up to index 5
1214 02:59:33.577305 (+524990us) log.cc:844] Serialized 14 byte log entry
1214 02:59:34.017270 (+439965us) log.cc:844] Serialized 14 byte log entry
1214 02:59:34.019467 (+ 2197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:34.019527 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:34.019573 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.019648 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.019789 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54723,"prepare.queue_time_us":204394,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":37,"raft.queue_time_us":73113,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":16135,"thread_start_us":231113,"threads_started":1}
W20251214 02:59:34.039264 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 219574us
I20251214 02:59:34.040994 23391 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:34.047521 21173 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:34.048425 23129 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:34.049628 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 298163us
I20251214 02:59:34.052079 23459 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:34.052625 23459 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.053632 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.055349 23459 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.056802 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.057229 23459 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.057509 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.058395 23459 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.060427 23459 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.065034 21173 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:33.962634 23444 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.063935 21096 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:34.067139 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 245532us
W20251214 02:59:34.066669 21126 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.001s sys 0.015s
W20251214 02:59:34.069186 21126 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.001s sys 0.015s
I20251214 02:59:33.988888 23422 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:33.971750 23263 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.072769 21096 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:33.980756 23392 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.060563 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.074066 23459 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:34.066870 23360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.001s sys 0.032s
I20251214 02:59:34.099761 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.103961 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.104616 22017 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:34.107317 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.006745 23435 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.113838 21282 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:34.115185 21282 raft_consensus.cc:606] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:33.988109 23492 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.127252 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:34.127509 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.074081 23221 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:34.100313 23360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.001s sys 0.032s
I20251214 02:59:34.132512 23393 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:34.149773 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.132691 23257 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
W20251214 02:59:34.154182 23491 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20251214 02:59:34.098251 23402 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.098102 23405 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.076298 23247 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.077431 23430 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.072309 23504 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:34.072487 23505 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:34.138880 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 66789.1us
I20251214 02:59:34.074498 23396 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.100796 23478 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:33.897100 23412 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:33.932586 23474 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.113185 23408 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.212440 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:34.220278 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 433074us
W20251214 02:59:34.224197 23274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.001s sys 0.013s
W20251214 02:59:34.224581 23274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s user 0.001s sys 0.013s
I20251214 02:59:34.224730 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:34.227448 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.232887 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:34.236235 21133 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.011s sys 0.005s
W20251214 02:59:34.236575 21133 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.742s user 0.011s sys 0.005s
I20251214 02:59:34.240804 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:34.251132 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.258317 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.292636 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.262799 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.274919 21128 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:34.275214 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.293661 21208 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:34.293618 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.294657 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.278656 21282 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75c047f1d6d94237afc050db2241d659"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
W20251214 02:59:34.294765 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 477488us
I20251214 02:59:34.238185 23466 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.249238 23393 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:34.296527 21282 raft_consensus.cc:606] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:34.290632 23412 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.294654 21127 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:34.305426 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.278149 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:34.299352 23480 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.217762 23503 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:34.266268 23392 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.303736 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.306005 21127 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:34.268438 23395 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.291162 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.228755 23396 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.221555 23405 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.222445 23247 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:34.348873 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 70826.6us
I20251214 02:59:34.281953 23510 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:34.228468 23430 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:34.356513 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.382073 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:34.383713 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:34.353075 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 524948us
I20251214 02:59:34.350317 21099 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:34.383864 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.250047 23408 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:34.391304 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.402393 23508 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.264474 23502 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.412832 23461 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.308079 23421 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.300606 23504 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.433598 23497 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:34.307343 23402 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.476051 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.381842 23480 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:34.469900 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:34.471515 23489 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:34.308413 23224 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.302742 23393 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:34.352741 23164 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:34.352392 23459 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 02:59:34.573426 21984 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.007s sys 0.020s
W20251214 02:59:34.573848 21984 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.007s sys 0.020s
I20251214 02:59:34.518847 23492 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.519098 23497 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:34.519835 23396 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.520584 23405 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.522950 23430 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:34.525338 23445 consensus_meta.cc:287] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.580s user 0.008s sys 0.000s
I20251214 02:59:34.525872 23505 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.390252 23392 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:34.415638 23173 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:34.246516 23478 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.307119 23466 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.417960 23510 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.562395 23393 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:34.580109 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.605264 23511 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:34.605970 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 222344us
I20251214 02:59:34.527137 23508 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.607550 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.582854 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:34.518724 23402 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.353271 23503 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.389516 23395 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.590240 23475 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.687448 21099 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:34.562181 23515 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:34.574445 23164 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.614202 23497 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:34.628865 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:34.662624 22018 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:34.562462 23480 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.648902 23445 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:34.658545 23493 log.cc:927] Time spent T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20251214 02:59:34.690704 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.649554 23112 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.568301 23504 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.652197 23505 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.654487 23173 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:34.699052 23247 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.482875 23502 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.704226 21275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2492) took 1391 ms. Trace:
I20251214 02:59:34.654187 23392 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.706202 23263 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.706116 23408 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.706681 23503 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.706939 23414 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:34.708048 23412 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:34.717265 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47078 (request call id 1136) took 1209 ms. Trace:
I20251214 02:59:34.742543 21129 rpcz_store.cc:276] 1214 02:59:33.507706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.787254 (+279548us) service_pool.cc:225] Handling call
1214 02:59:34.717080 (+929826us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001360d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:34.717203 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":274779392}
I20251214 02:59:34.708392 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.716778 21203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2492) took 1281 ms. Trace:
I20251214 02:59:34.743769 21203 rpcz_store.cc:276] 1214 02:59:33.435010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.435685 (+ 675us) service_pool.cc:225] Handling call
1214 02:59:33.755719 (+320034us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.265794 (+510075us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:34.265826 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:34.265865 (+ 39us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:34.265895 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.716562 (+450667us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400146ed4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.716609 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.716724 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1172733184}
I20251214 02:59:34.744867 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.745272 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.723682 23173 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:34.746949 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42016 (request call id 843) took 3987 ms. Trace:
I20251214 02:59:34.661681 23511 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.704133 23288 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.726248 21275 rpcz_store.cc:276] 1214 02:59:33.313254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.313362 (+ 108us) service_pool.cc:225] Handling call
1214 02:59:33.313614 (+ 252us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.426769 (+113155us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.426802 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.426847 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
1214 02:59:33.426877 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.704000 (+1277123us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540005a354. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.704051 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.704171 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259787520}
I20251214 02:59:34.722009 23112 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.746922 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.702948 21208 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 02:59:34.747885 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.747505 21206 rpcz_store.cc:276] 1214 02:59:30.752504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:30.755389 (+ 2885us) service_pool.cc:225] Handling call
1214 02:59:33.164374 (+2408985us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540011c720. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef1b550 00007f670ef982d2 00007f6713f5613a 00007f670d4f1934 00007f670d4f18c2 00007f670d4f17f1 00007f670d4f176c 00007f670d4f0a22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.739859 (+1575485us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b54001464d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef984d3 00007f6713f5613a 00007f670d4f1934 00007f670d4f18c2 00007f670d4f17f1 00007f670d4f176c 00007f670d4f0a22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:34.739960 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1981739136}
I20251214 02:59:34.736011 23474 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.773969 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.741354 23478 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.787209 22012 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:34.789412 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.704339 23475 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:34.794250 22014 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44274 (request call id 479) took 2045 ms (client timeout 1521 ms). Trace:
I20251214 02:59:34.797073 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:34.794742 22014 rpcz_store.cc:269] 1214 02:59:32.741795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:32.742548 (+ 753us) service_pool.cc:225] Handling call
1214 02:59:34.786773 (+2044225us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b540052d354. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:34.786887 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":132184448}
I20251214 02:59:34.796554 23510 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.799163 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.704790 23441 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.801694 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.803498 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.771309 23288 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:34.809702 22012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 554) took 1413 ms. Trace:
I20251214 02:59:34.813692 22012 rpcz_store.cc:276] 1214 02:59:33.396364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.396497 (+ 133us) service_pool.cc:225] Handling call
1214 02:59:34.786998 (+1390501us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.794337 (+ 7339us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.809262 (+ 14925us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540052d350. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:34.809624 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3197366656}
I20251214 02:59:34.745303 23521 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.815068 21208 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:34.817721 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47078 (request call id 1138) took 1309 ms. Trace:
I20251214 02:59:34.818150 21128 rpcz_store.cc:276] 1214 02:59:33.508177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.885519 (+377342us) service_pool.cc:225] Handling call
1214 02:59:34.817534 (+932015us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400031454. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:34.817654 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":808169344}
I20251214 02:59:34.746634 23414 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:34.807981 23483 log.cc:927] Time spent T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.288s user 0.005s sys 0.021s
I20251214 02:59:34.696795 23515 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.820724 23504 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.825031 21208 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:34.820919 21099 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:34.830684 23395 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:34.832598 23296 consensus_meta.cc:287] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.825s user 0.011s sys 0.000s
W20251214 02:59:34.833520 21133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.019s sys 0.048s
I20251214 02:59:34.841315 21274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 269) took 1819 ms. Trace:
W20251214 02:59:34.842742 21133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s user 0.020s sys 0.048s
W20251214 02:59:34.843159 21248 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.013s sys 0.055s
W20251214 02:59:34.843498 21248 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.013s sys 0.055s
I20251214 02:59:34.832639 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.846951 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:34.848165 21208 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
I20251214 02:59:34.850183 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:34.841892 21274 rpcz_store.cc:276] 1214 02:59:33.021573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.021927 (+ 354us) service_pool.cc:225] Handling call
1214 02:59:33.022176 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.153116 (+130940us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.153168 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.153211 (+ 43us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:33.153242 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.841027 (+1687785us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54001d4a54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.841072 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.841192 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300706560}
I20251214 02:59:34.850837 21277 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:34.804365 23508 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.854851 23475 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.866099 21099 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:34.885226 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.892133 21208 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:34.862758 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:34.895236 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:34.859234 21275 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:34.902843 21208 raft_consensus.cc:2410] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251214 02:59:34.913409 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.853607 23466 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.917398 21204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2479) took 1899 ms. Trace:
I20251214 02:59:34.826736 23502 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.912043 23407 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:34.852140 23394 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20251214 02:59:34.923049 23407 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:34.925032 23407 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.915748 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.926210 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.927210 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.928052 23112 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.928736 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.929795 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.929911 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.856056 23444 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:34.915019 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.855754 23164 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.917879 21204 rpcz_store.cc:276] 1214 02:59:33.017543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.017719 (+ 176us) service_pool.cc:225] Handling call
1214 02:59:33.037011 (+ 19292us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:33.093758 (+ 56747us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.093792 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:33.094301 (+ 509us) log.cc:844] Serialized 26 byte log entry
1214 02:59:33.275835 (+181534us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:33.275874 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.307467 (+1031593us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b54001ca554. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.307517 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:34.801611 (+494094us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:34.917187 (+115576us) raft_consensus.cc:1685] finished
1214 02:59:34.917216 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.917342 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":29771,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3024,"spinlock_wait_cycles":89513728,"thread_start_us":181139,"threads_started":1,"wal-append.queue_time_us":311917}
I20251214 02:59:34.853112 23402 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:34.923341 23505 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:34.824682 23521 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:35.041936 21171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.014s sys 0.023s
I20251214 02:59:36.133859 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42032 (request call id 1238) took 1245 ms. Trace:
W20251214 02:59:35.809829 21274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.013s sys 0.031s
W20251214 02:59:34.945031 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.031s sys 0.046s
I20251214 02:59:35.302117 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:34.941299 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:34.998728 21281 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2495) took 1677 ms. Trace:
I20251214 02:59:35.002040 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:35.083389 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.009s sys 0.041s
W20251214 02:59:35.001521 21125 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47062 (request call id 1778) took 1678 ms (client timeout 1574 ms). Trace:
W20251214 02:59:36.138494 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.878s user 0.013s sys 0.087s
W20251214 02:59:36.138787 21274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.228s user 0.017s sys 0.045s
I20251214 02:59:34.910921 23480 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:35.299124 21124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 235) took 1971 ms. Trace:
W20251214 02:59:35.030297 23214 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.025s sys 0.006s
I20251214 02:59:36.135058 21208 rpcz_store.cc:276] 1214 02:59:34.888918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.914797 (+ 25879us) service_pool.cc:225] Handling call
1214 02:59:36.133772 (+1218975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:35.018891 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:36.136520 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.879s user 0.049s sys 0.096s
I20251214 02:59:36.133641 21282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43308 (request call id 13) took 1859 ms. Trace:
W20251214 02:59:36.134356 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:36.148669 21282 rpcz_store.cc:276] 1214 02:59:34.274081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.274298 (+ 217us) service_pool.cc:225] Handling call
1214 02:59:35.061668 (+787370us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5400233d20. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef1b550 00007f670ef982d2 00007f6713f5613a 00007f670d4f1934 00007f670d4f18c2 00007f670d4f17f1 00007f670d4f176c 00007f670d4f0a22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:36.133562 (+1071894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9645824}
W20251214 02:59:36.133785 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.748s user 0.038s sys 0.058s
W20251214 02:59:35.730942 21247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.380s user 0.014s sys 0.066s
W20251214 02:59:36.149561 21247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.798s user 0.017s sys 0.070s
W20251214 02:59:36.025413 21126 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.329s user 0.000s sys 0.021s
I20251214 02:59:36.138062 21203 raft_consensus.cc:2376] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6. Re-sending same reply.
W20251214 02:59:36.149948 21126 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.453s user 0.000s sys 0.023s
I20251214 02:59:36.150225 21203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42032 (request call id 1245) took 1230 ms. Trace:
W20251214 02:59:35.731194 21279 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.342s user 0.026s sys 0.053s
W20251214 02:59:35.075273 21132 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.033s sys 0.089s
I20251214 02:59:36.150627 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:36.151312 21132 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.924s user 0.058s sys 0.111s
I20251214 02:59:36.151441 22012 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
W20251214 02:59:36.150720 21279 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.765s user 0.027s sys 0.054s
I20251214 02:59:36.139560 21124 rpcz_store.cc:276] 1214 02:59:33.327427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.327648 (+ 221us) service_pool.cc:225] Handling call
1214 02:59:33.332823 (+ 5175us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.507905 (+175082us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.507940 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.507992 (+ 52us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:33.508021 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:35.298842 (+1790821us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540018fe54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:35.298894 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:35.299054 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402198272}
W20251214 02:59:36.139832 21125 rpcz_store.cc:269] 1214 02:59:33.322987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.323103 (+ 116us) service_pool.cc:225] Handling call
1214 02:59:34.997673 (+1674570us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400314f54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:35.001437 (+ 3764us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1124770176}
I20251214 02:59:36.150542 21203 rpcz_store.cc:276] 1214 02:59:34.920276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.930915 (+ 10639us) service_pool.cc:225] Handling call
1214 02:59:36.150171 (+1219256us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251214 02:59:34.984926 21245 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.015s sys 0.013s
I20251214 02:59:34.911234 23445 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:36.137157 21207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2501) took 2040 ms (client timeout 2022 ms). Trace:
I20251214 02:59:36.137909 21281 rpcz_store.cc:276] 1214 02:59:33.321712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.321866 (+ 154us) service_pool.cc:225] Handling call
1214 02:59:33.322115 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.921757 (+599642us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.921786 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.921826 (+ 40us) raft_consensus.cc:1645] Marking committed up to 1
1214 02:59:33.921863 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.998464 (+1076601us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54001d5454. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:34.998511 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:34.998662 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1378746112}
I20251214 02:59:34.912161 23441 raft_consensus.cc:2804] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:36.150645 21126 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1792) took 1991 ms. Trace:
I20251214 02:59:36.139684 21274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1796) took 1281 ms. Trace:
W20251214 02:59:36.153963 21245 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.857s user 0.028s sys 0.055s
W20251214 02:59:36.174060 21207 rpcz_store.cc:269] 1214 02:59:34.097124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.098312 (+ 1188us) service_pool.cc:225] Handling call
1214 02:59:35.301954 (+1203642us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f53bd0 00007f6713f534b4 00007f670d4e3834 00007f670d4e37c2
1214 02:59:36.137080 (+835126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4968704}
I20251214 02:59:36.174875 21200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42032 (request call id 1247) took 1190 ms. Trace:
W20251214 02:59:36.184979 21277 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.237s user 0.004s sys 0.024s
W20251214 02:59:36.154479 21096 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.860s user 0.027s sys 0.046s
W20251214 02:59:36.187803 21277 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.252s user 0.004s sys 0.024s
W20251214 02:59:36.187907 21096 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.893s user 0.027s sys 0.046s
I20251214 02:59:36.184123 21126 rpcz_store.cc:276] 1214 02:59:34.158923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.528110 (+369187us) service_pool.cc:225] Handling call
1214 02:59:34.592226 (+ 64116us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.592447 (+ 221us) raft_consensus.cc:1534] Early marking committed up to index 6
1214 02:59:34.696423 (+103976us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.150317 (+1453894us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.150365 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:36.150413 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.150473 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.150595 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103627,"prepare.queue_time_us":104750,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":1453590,"threads_started":1}
W20251214 02:59:34.944455 21273 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.079s sys 0.185s
W20251214 02:59:36.188879 21273 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.130s user 0.110s sys 0.215s
W20251214 02:59:35.362669 23522 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.418s user 0.001s sys 0.000s
W20251214 02:59:36.165256 21171 connection.cc:232] RPC call timeout handler was delayed by -1.5696884070000579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20251214 02:59:35.185106 23525 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.184180 21274 rpcz_store.cc:276] 1214 02:59:34.858050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.859716 (+ 1666us) service_pool.cc:225] Handling call
1214 02:59:34.860029 (+ 313us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.860236 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 4
1214 02:59:34.910804 (+ 50568us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.139353 (+1228549us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.139405 (+ 52us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:36.139447 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.139513 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.139633 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":141626,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":8223,"raft.queue_time_us":89,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":1228171,"threads_started":1}
W20251214 02:59:35.826483 21985 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.223s user 0.001s sys 0.021s
W20251214 02:59:35.949916 23292 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.234s user 0.022s sys 0.038s
I20251214 02:59:36.185568 21200 rpcz_store.cc:276] 1214 02:59:34.984514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.991763 (+ 7249us) service_pool.cc:225] Handling call
1214 02:59:36.174812 (+1183049us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251214 02:59:36.188292 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 1.9339e+06us
I20251214 02:59:36.175482 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:35.401643 23414 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.162593 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:35.081152 23288 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:35.061050 23515 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:35.796360 23173 consensus_meta.cc:287] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 1.039s user 0.006s sys 0.003s
I20251214 02:59:35.009509 23392 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.211645 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2491) took 2898 ms. Trace:
I20251214 02:59:36.211999 21278 rpcz_store.cc:276] 1214 02:59:33.312981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.313092 (+ 111us) service_pool.cc:225] Handling call
1214 02:59:33.313345 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.488699 (+175354us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.488733 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.488778 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:33.488811 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.211429 (+2722618us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400046fd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:36.211476 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.211592 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402806016}
W20251214 02:59:36.200505 23292 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.484s user 0.024s sys 0.041s
I20251214 02:59:35.346233 23413 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:36.184463 21276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.165s user 0.012s sys 0.010s
W20251214 02:59:36.148386 23214 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.793s user 0.025s sys 0.033s
I20251214 02:59:34.850553 23296 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:35.281424 23475 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:34.943375 23437 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251214 02:59:36.248055 21276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.246s user 0.014s sys 0.018s
I20251214 02:59:36.199143 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:36.199753 21985 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.597s user 0.006s sys 0.022s
I20251214 02:59:36.153926 23402 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:35.083602 23511 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:34.985893 23438 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.154917 23480 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.206466 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.205610 22012 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 02:59:36.224465 22019 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44282 (request call id 553) took 2866 ms (client timeout 1607 ms). Trace:
I20251214 02:59:36.183920 23441 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:36.221603 23525 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:36.243553 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 1.30237e+06us
I20251214 02:59:35.665874 23407 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:36.223834 23392 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.244427 23292 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:36.257427 23437 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:36.245124 21983 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.006s sys 0.018s
I20251214 02:59:36.270849 21027 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:36.218874 23173 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.282231 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1233) took 1988 ms. Trace:
I20251214 02:59:36.282599 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 231) took 2956 ms. Trace:
W20251214 02:59:36.241245 21986 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.002s sys 0.010s
I20251214 02:59:36.251646 23485 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:36.281019 22019 rpcz_store.cc:269] 1214 02:59:33.345858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.345986 (+ 128us) service_pool.cc:225] Handling call
1214 02:59:36.211749 (+2865763us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b5400792cd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.211857 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58051968}
I20251214 02:59:36.284725 23441 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:36.255875 23541 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:36.242653 23413 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.253764 23475 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:36.252310 23497 consensus_meta.cc:287] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 1.554s user 0.006s sys 0.009s
I20251214 02:59:36.282112 23401 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:36.293268 21206 rpcz_store.cc:276] 1214 02:59:34.293635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.814400 (+520765us) service_pool.cc:225] Handling call
1214 02:59:34.823922 (+ 9522us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.824095 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:34.824119 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:34.824160 (+ 41us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:34.824187 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.824236 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.282062 (+1457826us) spinlock_profiling.cc:243] Waited 8.42 ms on lock 0x7b540008fa50. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.282179 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19359616}
I20251214 02:59:36.254873 23402 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.307260 23401 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.309180 23401 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.309333 23402 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.310030 23402 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 02:59:36.311000 23393 consensus_meta.cc:287] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 1.621s user 0.011s sys 0.001s
I20251214 02:59:36.311520 23393 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:36.312639 23528 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251214 02:59:36.292496 23292 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.310972 23296 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.310469 23402 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.314888 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1789) took 2156 ms. Trace:
I20251214 02:59:36.315284 21131 rpcz_store.cc:276] 1214 02:59:34.158033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.159284 (+ 1251us) service_pool.cc:225] Handling call
1214 02:59:34.819331 (+660047us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.819586 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:34.819616 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:34.819660 (+ 44us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:34.819693 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:34.819754 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.314733 (+1494979us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400031450. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.314834 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1517409024}
I20251214 02:59:36.317020 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 237) took 2989 ms. Trace:
I20251214 02:59:36.318297 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1232) took 2034 ms. Trace:
W20251214 02:59:36.297108 21986 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.626s user 0.004s sys 0.011s
W20251214 02:59:36.292093 21983 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.661s user 0.006s sys 0.019s
I20251214 02:59:36.290345 23437 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:36.277136 23438 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.293336 21133 rpcz_store.cc:276] 1214 02:59:33.326198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.326328 (+ 130us) service_pool.cc:225] Handling call
1214 02:59:33.343751 (+ 17423us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.421206 (+ 77455us) raft_consensus.cc:1534] Early marking committed up to index 8
1214 02:59:34.236916 (+815710us) log.cc:844] Serialized 14 byte log entry
1214 02:59:34.854382 (+617466us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:34.854434 (+ 52us) raft_consensus.cc:1645] Marking committed up to 8
1214 02:59:34.854484 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.282334 (+1427850us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b54002bd754. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:36.282383 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.282516 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15128,"prepare.queue_time_us":85795,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":14085,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":177662720,"thread_start_us":1348666,"threads_started":2,"wal-append.queue_time_us":1138428}
I20251214 02:59:36.292542 23392 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.326161 23525 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.327142 23392 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.306105 23485 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.328799 23435 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.329612 23485 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.330885 23173 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.331144 21280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 273) took 1457 ms. Trace:
I20251214 02:59:36.331599 21280 rpcz_store.cc:276] 1214 02:59:34.873637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.919528 (+ 45891us) service_pool.cc:225] Handling call
1214 02:59:35.380040 (+460512us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:35.380223 (+ 183us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:35.380249 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:35.380290 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:35.380326 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:35.380380 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.330970 (+950590us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540007e4d0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.331082 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1058518912}
W20251214 02:59:36.335994 23518 log.cc:927] Time spent T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251214 02:59:36.337379 23475 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.339744 23413 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.343921 21276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1798) took 1475 ms. Trace:
I20251214 02:59:36.318670 21201 rpcz_store.cc:276] 1214 02:59:34.284186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.803770 (+519584us) service_pool.cc:225] Handling call
1214 02:59:34.804090 (+ 320us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:36.187618 (+1383528us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:36.187650 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.187694 (+ 44us) raft_consensus.cc:1645] Marking committed up to 2
1214 02:59:36.187721 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.318083 (+130362us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540037a0d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:36.318137 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.318243 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3179395840}
I20251214 02:59:36.299079 23521 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.345242 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2498) took 2237 ms. Trace:
I20251214 02:59:36.320683 21127 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:36.290333 23407 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:36.319216 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 753495us
I20251214 02:59:36.313488 23393 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.317366 21130 rpcz_store.cc:276] 1214 02:59:33.328002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.329602 (+ 1600us) service_pool.cc:225] Handling call
1214 02:59:33.329852 (+ 250us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.831025 (+501173us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:33.831052 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:33.831088 (+ 36us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:33.831116 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.316808 (+2485692us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54002bdc54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:36.316850 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.316964 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1152297216}
I20251214 02:59:36.344386 21276 rpcz_store.cc:276] 1214 02:59:34.868976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.872778 (+ 3802us) service_pool.cc:225] Handling call
1214 02:59:34.912637 (+ 39859us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:34.912984 (+ 347us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:34.913011 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:34.919980 (+ 6969us) log.cc:844] Serialized 26 byte log entry
1214 02:59:36.249298 (+1329318us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:36.249342 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.249413 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:36.343705 (+ 94292us) raft_consensus.cc:1685] finished
1214 02:59:36.343729 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.343861 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":451,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5338,"thread_start_us":1246809,"threads_started":1,"wal-append.queue_time_us":1320018}
I20251214 02:59:36.347218 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:36.354401 21126 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:36.347893 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.346143 21205 rpcz_store.cc:276] 1214 02:59:34.096281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.097792 (+ 1511us) service_pool.cc:225] Handling call
1214 02:59:36.329856 (+2232064us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:36.330068 (+ 212us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:36.330431 (+ 363us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:36.330471 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.330519 (+ 48us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:36.333509 (+ 2990us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.333598 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.333716 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12302,"spinlock_wait_cycles":5833088}
I20251214 02:59:36.352824 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.359012 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.355788 21201 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:36.359619 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:36.362035 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.365684 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.346840 23217 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.366266 21204 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:36.366374 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.366786 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.367790 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:36.392432 22012 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:36.368422 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:36.322494 23497 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.448491 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1791) took 2289 ms. Trace:
I20251214 02:59:36.456202 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.459820 21124 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
W20251214 02:59:36.459956 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 91200.3us
I20251214 02:59:36.459470 21132 rpcz_store.cc:276] 1214 02:59:34.158662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.159738 (+ 1076us) service_pool.cc:225] Handling call
1214 02:59:34.159985 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.160196 (+ 211us) raft_consensus.cc:1534] Early marking committed up to index 4
1214 02:59:34.226937 (+ 66741us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.448141 (+2221204us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.448194 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:36.448234 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.448302 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.448424 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":95005,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":132444,"spinlock_wait_cycles":5086592,"thread_start_us":2220750,"threads_started":2,"wal-append.queue_time_us":1996381}
I20251214 02:59:36.456367 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.299650 23480 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.361018 23475 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:36.470151 21202 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47944 (request call id 253) took 2675 ms (client timeout 2080 ms). Trace:
I20251214 02:59:36.470894 21126 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
W20251214 02:59:36.473418 21202 rpcz_store.cc:269] 1214 02:59:33.795099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.797022 (+ 1923us) service_pool.cc:225] Handling call
1214 02:59:36.469954 (+2672932us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540024b654. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.470062 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288543488}
I20251214 02:59:36.453712 23471 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.477138 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.477262 23471 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.453574 23414 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.478201 21200 raft_consensus.cc:2393] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 6: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:36.485894 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.492535 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.489217 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.492059 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.493839 21133 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:36.471587 21203 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:36.484571 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.483664 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.495247 21124 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:36.495384 21126 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:36.496132 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.496295 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.496867 21126 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:36.497753 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.498478 21126 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:36.478535 23414 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.373407 23465 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.504361 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.524046 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.524883 23414 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.488916 23475 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.488804 23438 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.487972 23480 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.526782 23402 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.535292 23546 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:36.553539 23546 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.558645 23546 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:36.528878 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 55760.4us
I20251214 02:59:36.529913 21130 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:36.548015 23465 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:36.532275 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 339782us
W20251214 02:59:36.542408 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 76091us
I20251214 02:59:36.548514 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:36.549296 21247 connection.cc:232] RPC call timeout handler was delayed by -1.0927752199999077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20251214 02:59:36.493417 21125 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:36.556376 23441 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.490586 23407 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:36.530586 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 373238us
I20251214 02:59:36.571930 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.575472 21986 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.569620 21132 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:36.491070 23292 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.454704 23217 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.480288 23497 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.558393 23479 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.605177 23479 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:36.606003 23217 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.574785 23480 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.607317 23480 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.592496 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:36.594470 21127 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52098 (request call id 244) took 2180 ms (client timeout 1587 ms). Trace:
I20251214 02:59:36.608317 21199 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 9
I20251214 02:59:36.609151 23480 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.610741 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.608618 21131 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:36.594584 21129 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:36.615908 23441 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.616279 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 1143) took 2395 ms. Trace:
I20251214 02:59:36.616640 21129 rpcz_store.cc:276] 1214 02:59:34.221245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.756614 (+535369us) service_pool.cc:225] Handling call
1214 02:59:36.594358 (+1837744us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:36.615146 (+ 20788us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.616109 (+ 963us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b5400169750. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.616225 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4223575808}
I20251214 02:59:36.617260 21173 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.596514 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.603472 23475 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.603801 23511 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251214 02:59:36.608297 21127 rpcz_store.cc:269] 1214 02:59:34.414283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.415926 (+ 1643us) service_pool.cc:225] Handling call
1214 02:59:36.594190 (+2178264us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400169754. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.594383 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3757611904}
I20251214 02:59:36.621598 21199 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 02:59:36.601704 23437 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:36.622398 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47028.1us
W20251214 02:59:36.623294 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 51449.9us
I20251214 02:59:36.623296 21096 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:36.624444 21985 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.624897 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.628824 21276 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:36.629534 21247 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.629750 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:36.630481 21280 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:36.630858 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.631588 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.631517 21127 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:36.632210 21276 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:36.633215 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:36.625651 21199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2499) took 2529 ms (client timeout 2270 ms). Trace:
I20251214 02:59:36.633771 21278 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:36.632157 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.634965 21985 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.635855 21247 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:36.636344 21199 rpcz_store.cc:269] 1214 02:59:34.096567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.098056 (+ 1489us) service_pool.cc:225] Handling call
1214 02:59:36.592311 (+2494255us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b5800002798. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f53bd0 00007f6713f534b4 00007f670d4e3834 00007f670d4e37c2
1214 02:59:36.625587 (+ 33276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5430016}
I20251214 02:59:36.637054 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.638906 21171 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.640633 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.637213 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.641645 21096 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:36.638236 21247 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:36.644200 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.647045 21278 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:36.645641 21132 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:36.622072 23557 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.624892 23193 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.665922 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 02:59:36.666653 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 49494.5us
I20251214 02:59:36.666944 21096 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:36.648695 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.607867 23479 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:36.664373 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.670938 21278 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:36.645013 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.627703 23480 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:36.664137 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:36.624609 23511 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:36.627492 23461 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:36.634860 23546 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:36.640991 23401 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:36.643028 23552 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:36.670842 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.642935 23489 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:36.643244 23475 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:36.720060 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.673664 21171 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.641317 23257 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:36.744709 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.645622 23538 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:36.744721 21277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1795) took 1889 ms. Trace:
I20251214 02:59:36.747107 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.806164 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:36.793166 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 49330.8us
I20251214 02:59:36.665449 23217 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:36.788614 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 124536us
I20251214 02:59:36.666095 23292 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:36.673115 23278 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:36.720606 23360 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:36.828001 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.922892 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.955514 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.827672 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.956132 21276 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:36.956327 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.672988 23545 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:36.816242 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.643651 23559 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:36.827183 23393 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:36.742877 23511 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:36.746194 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 78428.1us
I20251214 02:59:36.670199 23497 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:36.929618 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:36.925354 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 251764us
I20251214 02:59:36.928838 23465 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:36.670845 23557 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.924074 21247 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.941000 22020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44272 (request call id 356) took 3097 ms. Trace:
I20251214 02:59:36.954577 23360 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:36.957795 23480 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:36.955614 21203 raft_consensus.cc:2376] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9. Re-sending same reply.
I20251214 02:59:36.963016 23551 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:36.988001 22020 rpcz_store.cc:276] 1214 02:59:33.843900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.853548 (+ 9648us) service_pool.cc:225] Handling call
1214 02:59:36.211877 (+2358329us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:36.212055 (+ 178us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:36.212077 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.212125 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:36.212158 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.212208 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.940794 (+728586us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b5400792cd0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:36.940912 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5423419648}
I20251214 02:59:36.743093 23461 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:36.930806 23240 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:36.967136 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:36.968006 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.969007 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.964238 21280 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:36.923946 21278 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:36.969583 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.669648 23193 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.969542 23479 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.973652 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.943411 23437 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:36.832527 23475 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:36.925608 23538 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.969542 23401 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.742617 23529 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.228s
I20251214 02:59:36.952175 23217 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.970119 23489 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:36.979027 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.980005 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.975226 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:36.989368 23559 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:36.985265 21171 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:36.984483 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.015810 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.673985 23550 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:36.987150 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.815239 21206 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:36.984858 21027 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 3 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:37.012629 23538 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:36.964017 23545 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:36.994766 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.995322 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.996794 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.025414 21274 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:37.026345 21985 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:36.994467 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:36.998286 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.998935 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:36.767406 23546 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.000223 21204 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:37.000610 21173 leader_election.cc:400] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9.
I20251214 02:59:37.015193 23401 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:36.970299 23393 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:36.991930 23360 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.022630 23489 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.036708 23401 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:37.013906 21282 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
W20251214 02:59:37.015362 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:36.923599 21277 rpcz_store.cc:276] 1214 02:59:34.855060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:34.857657 (+ 2597us) service_pool.cc:225] Handling call
1214 02:59:34.914138 (+ 56481us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.914344 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 6
1214 02:59:36.188342 (+1273998us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.744397 (+556055us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.744454 (+ 57us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:36.744503 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.744563 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:36.744669 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71583,"prepare.queue_time_us":166064,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":72105,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":38720,"thread_start_us":1736285,"threads_started":2}
I20251214 02:59:37.006096 23479 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.016762 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:36.993912 23461 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:37.018669 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:37.019929 21171 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:36.988380 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 64402.6us
I20251214 02:59:37.041358 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:37.023551 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.042457 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.043275 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:37.043465 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:37.043903 22020 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:37.027902 21203 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:37.017088 23214 raft_consensus.cc:3055] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:36.993351 23240 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:37.030220 21173 leader_election.cc:403] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:37.045526 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:37.045795 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45294us
I20251214 02:59:37.031001 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:37.021082 23482 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:37.022316 23529 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:37.021696 23550 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.001937 21281 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:37.013969 21200 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:36.998781 23551 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.028367 23545 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:37.064682 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:37.065392 21132 raft_consensus.cc:2393] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:37.066699 23475 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.067744 23479 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:37.068918 23485 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:37.070199 23193 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:37.071386 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 300) took 7623 ms. Trace:
I20251214 02:59:37.045969 23214 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:37.052620 23240 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:37.052345 23401 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:37.062108 23480 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:37.075068 23480 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.075884 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.076086 23217 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:37.059224 23529 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.076881 23529 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:37.077636 23214 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.079357 23556 log.cc:927] Time spent T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.096s user 0.002s sys 0.000s
I20251214 02:59:37.080128 21027 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 7 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:37.081724 21027 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:37.083307 21027 catalog_manager.cc:5654] T 75e354ca56944ff1b7775a8309bca568 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251214 02:59:37.083969 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 455849us
I20251214 02:59:37.030371 23407 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:37.085144 23407 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:37.085271 21986 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:37.086071 23407 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.092828 21279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1792) took 3756 ms. Trace:
I20251214 02:59:37.093145 21204 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:37.094332 21027 catalog_manager.cc:5654] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:37.095336 21124 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:37.086444 21986 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:37.044699 21202 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:37.048614 21199 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:37.101704 21099 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:37.031894 23457 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:37.052400 23129 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:37.053383 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.056263 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2521) took 2128 ms. Trace:
I20251214 02:59:37.056845 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.102321 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.037438 23393 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.062968 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.064021 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:37.115278 21173 connection.cc:232] RPC call timeout handler was delayed by -1.539559881999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20251214 02:59:37.115675 21173 connection.cc:232] RPC call timeout handler was delayed by -1.0553598819999479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20251214 02:59:37.064172 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:37.053866 23559 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:37.071770 21030 rpcz_store.cc:276] 1214 02:59:29.447627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:29.448115 (+ 488us) service_pool.cc:225] Handling call
1214 02:59:37.071302 (+7623187us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:31.135899 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:31.242723 (+106824us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:31.245495 (+ 2772us) write_op.cc:435] Acquired schema lock
1214 02:59:31.253271 (+ 7776us) tablet.cc:660] Decoding operations
1214 02:59:31.346930 (+ 93659us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:31.382583 (+ 35653us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:31.387271 (+ 4688us) tablet.cc:683] Acquiring locks for 2 operations
1214 02:59:31.466662 (+ 79391us) tablet.cc:699] Row locks acquired
1214 02:59:31.466684 (+ 22us) write_op.cc:260] PREPARE: finished
1214 02:59:31.466843 (+ 159us) write_op.cc:270] Start()
1214 02:59:31.466956 (+ 113us) write_op.cc:276] Timestamp: P: 1765681171466794 usec, L: 0
1214 02:59:31.466974 (+ 18us) op_driver.cc:348] REPLICATION: starting
1214 02:59:31.467552 (+ 578us) log.cc:844] Serialized 929 byte log entry
1214 02:59:31.707062 (+239510us) op_driver.cc:464] REPLICATION: finished
1214 02:59:33.136726 (+1429664us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f67129d6e40
1214 02:59:33.814625 (+677899us) write_op.cc:301] APPLY: starting
1214 02:59:33.818629 (+ 4004us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:33.856096 (+ 37467us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:33.857897 (+ 1801us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:33.932086 (+ 74189us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:33.959785 (+ 27699us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 02:59:33.965985 (+ 6200us) write_op.cc:312] APPLY: finished
1214 02:59:33.986755 (+ 20770us) log.cc:844] Serialized 29 byte log entry
1214 02:59:34.413161 (+426406us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:34.413813 (+ 652us) write_op.cc:454] Released schema lock
1214 02:59:34.413859 (+ 46us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:34.414291 (+ 432us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:34.414328 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6081,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":2052919,"apply.run_cpu_time_us":99872,"apply.run_wall_time_us":616448,"num_ops":2,"prepare.queue_time_us":306,"prepare.run_cpu_time_us":3796,"prepare.run_wall_time_us":331923,"replication_time_us":239941,"spinlock_wait_cycles":19491200,"thread_start_us":1956387,"threads_started":2,"wal-append.queue_time_us":1014786,"wal-append.run_cpu_time_us":1233,"wal-append.run_wall_time_us":1076560}]]}
I20251214 02:59:37.074272 21200 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:37.127840 21099 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:37.128697 23497 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:37.129201 23497 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:37.130040 23497 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.100594 22020 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:37.100076 21027 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 3 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:37.133755 21171 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:37.047870 21247 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.068610 23360 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.103601 21986 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:37.103436 23485 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:37.100946 21279 rpcz_store.cc:276] 1214 02:59:33.336601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.792798 (+456197us) service_pool.cc:225] Handling call
1214 02:59:33.793779 (+ 981us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:34.005176 (+211397us) raft_consensus.cc:1534] Early marking committed up to index 4
1214 02:59:34.385935 (+380759us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.574019 (+2188084us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.574091 (+ 72us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:36.574152 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:37.092602 (+518450us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540005b754. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:37.092646 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:37.092765 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":114415,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":101572,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":22414,"spinlock_wait_cycles":485579008,"thread_start_us":2488289,"threads_started":3,"wal-append.queue_time_us":1945923}
I20251214 02:59:37.051807 21206 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:37.116217 21130 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:37.117084 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:37.117712 21207 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:37.145808 21202 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:37.122697 23457 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:37.104343 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.136906 21203 raft_consensus.cc:1240] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
W20251214 02:59:37.136637 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 88841us
I20251214 02:59:37.110679 21208 rpcz_store.cc:276] 1214 02:59:34.927659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:36.148609 (+1220950us) service_pool.cc:225] Handling call
1214 02:59:36.148934 (+ 325us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:36.643842 (+494908us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:36.643871 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:36.644649 (+ 778us) log.cc:844] Serialized 26 byte log entry
1214 02:59:36.892705 (+248056us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:36.892736 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:36.892793 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:37.056078 (+163285us) raft_consensus.cc:1685] finished
1214 02:59:37.056097 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:37.056211 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":126161,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":168,"thread_start_us":618288,"threads_started":2,"wal-append.queue_time_us":345037}
I20251214 02:59:37.137884 22014 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:37.062780 23550 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:37.139925 21986 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:37.048178 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:37.114339 23129 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.109942 23445 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:37.136651 23559 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:37.161909 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 113791us
I20251214 02:59:37.163475 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.164222 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.166729 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.146286 21173 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:37.103063 23214 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.114430 23278 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:37.151891 21247 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.124804 23543 raft_consensus.cc:2804] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:37.166329 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.137282 23489 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:37.102764 23357 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9.
I20251214 02:59:37.170275 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.159224 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.124195 23450 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.140588 23401 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:37.169674 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.149174 23549 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:37.149318 23193 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:37.219487 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:37.142647 23475 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:37.241541 21273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1785) took 3906 ms. Trace:
I20251214 02:59:37.159143 23129 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:37.435019 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:37.437741 23240 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.440943 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:37.439834 21273 rpcz_store.cc:276] 1214 02:59:33.334887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:33.415860 (+ 80973us) service_pool.cc:225] Handling call
1214 02:59:33.416443 (+ 583us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:33.896569 (+480126us) raft_consensus.cc:1534] Early marking committed up to index 4
1214 02:59:34.058603 (+162034us) log.cc:844] Serialized 14 byte log entry
1214 02:59:36.529270 (+2470667us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:36.529323 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:36.529368 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:37.226666 (+697298us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54001fe0d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:37.241283 (+ 14617us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:37.241470 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":175967,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":122630,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":1103699200,"thread_start_us":2612512,"threads_started":3,"wal-append.queue_time_us":2184229,"wal-append.run_cpu_time_us":1759,"wal-append.run_wall_time_us":1158768}
I20251214 02:59:37.439657 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.170495 23559 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:37.175315 23349 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:37.168772 23456 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:37.403822 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:37.445436 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 293622us
I20251214 02:59:37.453806 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.455159 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.456233 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.179318 23515 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:37.179793 23557 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:37.218775 23401 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.430327 23546 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:37.459887 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.491653 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.402381 23529 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.458200 23456 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.322317 23479 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:37.296216 23543 raft_consensus.cc:697] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:37.446988 23349 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:37.319892 23461 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.179032 23278 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.424489 23480 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:37.535485 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.535595 21279 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:37.451401 23193 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:37.559993 23511 consensus_meta.cc:287] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.572s user 0.008s sys 0.000s
I20251214 02:59:37.536857 23401 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:37.228874 23450 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.457890 23515 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:37.623782 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.582033 23456 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:37.594720 23349 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.582461 23479 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.594025 23551 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:37.598173 23531 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:37.594213 23360 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:37.646636 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:37.663755 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:37.641952 23563 log.cc:927] Time spent T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251214 02:59:37.594961 23577 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:37.676654 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.624537 23240 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:37.683483 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:37.645112 23515 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:37.598940 23557 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:37.619156 23193 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.677795 23538 consensus_meta.cc:287] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.645s user 0.007s sys 0.001s
I20251214 02:59:37.650771 23479 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.700685 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:37.660214 23531 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.679654 23577 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.743923 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:37.775656 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 319497us
I20251214 02:59:37.637153 23511 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.815323 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.011s sys 0.010s
I20251214 02:59:37.815517 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
W20251214 02:59:37.816627 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.011s sys 0.010s
I20251214 02:59:37.813447 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:37.816880 22012 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:37.817209 22020 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:37.693814 23557 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.814437 21247 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:37.814751 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 279218us
W20251214 02:59:37.776415 23545 consensus_meta.cc:287] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.681s user 0.009s sys 0.000s
I20251214 02:59:37.810866 21282 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
W20251214 02:59:37.814360 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 251195us
W20251214 02:59:37.778311 23567 log.cc:927] Time spent T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.346s user 0.010s sys 0.000s
I20251214 02:59:37.795784 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.820399 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.811192 23531 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:37.823225 21203 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:37.778954 23485 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.834419 21247 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:37.820837 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 76994.7us
I20251214 02:59:37.843405 22020 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:37.844487 21171 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:37.813267 23577 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:37.781778 23360 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20251214 02:59:37.826077 23557 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:37.854704 22012 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:37.831614 23546 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:37.858985 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.835186 23552 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:37.857537 21171 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:37.812036 23538 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.873399 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:37.856531 23560 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:37.887646 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.901000 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.902727 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.903306 21202 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:37.903723 21207 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:37.911306 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 76969.1us
I20251214 02:59:37.834235 23545 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:37.858526 23579 log.cc:927] Time spent T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20251214 02:59:37.889659 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:37.811808 23543 consensus_queue.cc:237] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:37.900933 21279 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:37.941748 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:37.915417 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:37.944373 22012 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:37.944011 22020 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:37.913522 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:37.931708 21247 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.950958 21171 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 02:59:37.891361 23550 consensus_meta.cc:287] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.723s user 0.004s sys 0.005s
I20251214 02:59:37.866910 23340 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:37.877251 23221 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:37.869974 23552 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:37.967654 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:37.891654 23581 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:37.869688 23546 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 02:59:37.888610 23568 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.374s user 0.000s sys 0.000s
I20251214 02:59:37.924278 23529 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:37.964401 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:37.851539 23485 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:37.967285 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:37.978246 21279 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:37.971477 22018 raft_consensus.cc:1240] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:37.924108 23521 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:37.964496 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:37.976399 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:37.910908 23504 consensus_queue.cc:1059] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20251214 02:59:38.012320 23504 raft_consensus.cc:3055] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:37.923516 23461 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:37.949462 23450 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:38.015828 23576 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20251214 02:59:37.890820 23590 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20251214 02:59:38.010344 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:37.891847 23551 consensus_queue.cc:1059] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20251214 02:59:37.963784 23475 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:37.973204 23552 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:37.961863 23278 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:38.045949 21098 leader_election.cc:336] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.549s (SENT)
I20251214 02:59:38.047643 21098 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.031297 21127 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:38.031513 22012 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:37.973098 23581 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.020543 23521 raft_consensus.cc:697] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:38.009673 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:37.961625 23221 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:37.975967 23546 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:38.052105 23489 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:38.051554 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 100933us
I20251214 02:59:38.052606 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.012681 23504 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:38.071070 23521 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.071202 23504 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.072512 23504 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:38.072880 23393 consensus_meta.cc:287] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.935s user 0.007s sys 0.002s
I20251214 02:59:37.957868 23550 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.019449 23586 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:38.055226 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.059957 21096 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.062088 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:38.063216 21127 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:38.040881 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:38.064657 23574 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.471s user 0.000s sys 0.001s
I20251214 02:59:38.058085 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:38.057704 21098 leader_election.cc:336] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.553s (SENT)
I20251214 02:59:38.067363 22014 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 562) took 1006 ms. Trace:
I20251214 02:59:38.070295 23511 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.031329 23399 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:38.073870 21208 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20251214 02:59:38.076251 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 125374us
I20251214 02:59:38.098825 21171 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:38.054435 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:37.970456 23340 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.052425 22015 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:38.064467 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:38.092126 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:38.093075 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 179640us
I20251214 02:59:38.053089 23581 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:38.095739 22014 rpcz_store.cc:276] 1214 02:59:37.060813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.060979 (+ 166us) service_pool.cc:225] Handling call
1214 02:59:37.061263 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:37.152778 (+ 91515us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:38.067115 (+914337us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b540052d354. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:38.067297 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175794944}
I20251214 02:59:38.095701 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.078385 22019 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:38.100354 21124 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:38.097397 23399 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:38.109294 21281 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:38.109867 23545 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.079151 23485 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:38.095054 23217 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.114169 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.073267 23393 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:38.147503 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 89840.3us
I20251214 02:59:38.164709 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:38.165611 21098 leader_election.cc:336] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.571s (SENT)
I20251214 02:59:38.055962 23475 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.114069 23560 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:38.053551 23592 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
W20251214 02:59:38.151351 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 64808.5us
I20251214 02:59:38.086054 23586 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.110643 23596 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20251214 02:59:38.181213 23586 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:38.182168 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.155689 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.183313 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.183892 21275 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
W20251214 02:59:38.157122 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 82188.9us
I20251214 02:59:38.097676 23489 consensus_queue.cc:1059] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20251214 02:59:38.186709 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.187073 23489 raft_consensus.cc:3055] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:38.187456 23489 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:38.160903 23340 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:38.167836 21098 leader_election.cc:336] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.591s (SENT)
W20251214 02:59:38.172315 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:38.168788 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.195451 21098 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:38.178601 23589 log.cc:927] Time spent T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
W20251214 02:59:38.179311 23591 log.cc:927] Time spent T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
W20251214 02:59:38.179528 23456 consensus_meta.cc:287] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.532s user 0.007s sys 0.001s
I20251214 02:59:38.202272 23456 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.203365 23560 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.204177 23456 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:38.168694 23129 consensus_meta.cc:287] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.719s user 0.007s sys 0.003s
I20251214 02:59:38.180546 23607 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:38.070402 23221 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.189580 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.208593 21282 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 02:59:38.186072 21986 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:38.209874 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.210915 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.211534 21282 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:38.161662 23399 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:38.169131 23580 log.cc:927] Time spent T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.357s user 0.001s sys 0.000s
I20251214 02:59:38.196919 21277 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
W20251214 02:59:38.197713 21098 leader_election.cc:336] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.628s (SENT)
I20251214 02:59:38.199914 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:38.163868 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 73992.5us
I20251214 02:59:38.212555 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.189255 23538 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.213835 21275 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:38.213646 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.214008 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.215958 23607 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:38.217624 23616 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:38.218060 23616 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.205663 23545 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.209430 23129 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.219589 23393 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.196507 23593 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:38.220836 21200 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:38.223362 21098 leader_election.cc:336] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.605s (SENT)
I20251214 02:59:38.224128 21098 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.224960 23129 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.225118 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.225765 23604 raft_consensus.cc:3055] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:38.225932 23615 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.226218 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.227283 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.188297 23489 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.228192 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.228440 23489 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:38.215727 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.213110 23609 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:38.214476 23479 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.231595 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:38.232249 22016 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:38.223285 23616 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:38.232339 23475 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:38.218133 23214 consensus_meta.cc:287] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 1.041s user 0.010s sys 0.000s
I20251214 02:59:38.233373 21280 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:38.235057 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:38.234169 21098 leader_election.cc:336] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.635s (SENT)
I20251214 02:59:38.238855 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.239583 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.240422 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:38.258435 21203 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:38.214438 21245 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.213754 23587 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251214 02:59:38.285234 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:38.253504 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:38.266500 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 167153us
I20251214 02:59:38.268458 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.226163 23604 raft_consensus.cc:740] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:38.339479 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 104523us
I20251214 02:59:38.267035 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.295951 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:38.299482 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 85122us
I20251214 02:59:38.222856 23593 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:38.246608 23444 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:38.258790 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:38.231570 23609 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.229203 23515 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:38.301020 23559 consensus_meta.cc:287] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.841s user 0.009s sys 0.000s
I20251214 02:59:38.366154 21124 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
W20251214 02:59:38.334898 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 183908us
I20251214 02:59:38.335188 21273 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:38.328518 23357 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:38.223155 23545 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.292055 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:38.342109 21274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2520) took 1316 ms. Trace:
I20251214 02:59:38.342657 23584 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 02:59:38.343317 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 109182us
W20251214 02:59:38.380584 21098 leader_election.cc:336] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.708s (SENT)
I20251214 02:59:38.338928 23611 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.381054 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.346740 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.355953 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.382755 22014 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.383302 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.383261 22014 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:38.356460 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.384999 22012 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:38.218230 23607 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.293218 23217 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:38.268294 23550 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.258607 23393 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
W20251214 02:59:38.339411 23349 consensus_meta.cc:287] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.663s user 0.005s sys 0.004s
I20251214 02:59:38.351657 23593 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:38.372478 21274 rpcz_store.cc:276] 1214 02:59:37.025645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.025968 (+ 323us) service_pool.cc:225] Handling call
1214 02:59:37.026254 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:37.026469 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:37.026497 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:37.030588 (+ 4091us) log.cc:844] Serialized 26 byte log entry
1214 02:59:37.443140 (+412552us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:37.443178 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:37.443249 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:38.336471 (+893222us) raft_consensus.cc:1685] finished
1214 02:59:38.336495 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:38.342051 (+ 5556us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":388367,"threads_started":1,"wal-append.queue_time_us":614216}
I20251214 02:59:38.376998 21275 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:38.392256 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:38.393383 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.359246 22015 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:38.368113 23583 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:38.392354 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.334123 23557 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.360329 23609 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:38.386551 21096 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:38.341058 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:38.405622 22012 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
W20251214 02:59:38.453555 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 112619us
W20251214 02:59:38.427070 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57037.5us
I20251214 02:59:38.374812 23471 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:38.432574 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:38.431473 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:38.437458 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 50983.5us
I20251214 02:59:38.327351 23606 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20251214 02:59:38.452355 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.449437 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.374984 23559 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.459362 23559 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.460101 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.460789 21278 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:38.294936 23615 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.462790 23615 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:38.464510 23577 consensus_meta.cc:287] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.598s user 0.007s sys 0.008s
I20251214 02:59:38.464968 23577 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:38.467012 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:42016 (request call id 845) took 1523 ms. Trace:
I20251214 02:59:38.473019 21205 rpcz_store.cc:276] 1214 02:59:36.929210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:36.955486 (+ 26276us) service_pool.cc:225] Handling call
1214 02:59:38.452210 (+1496724us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54001464d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6713f5758e 00007f670d4f6437 00007f670d4f63c2 00007f670d4f62f1 00007f670d4f626c 00007f670d4f5522 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:38.452553 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":435668864}
I20251214 02:59:38.474522 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.475660 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.467319 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:38.378861 23357 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.428138 23217 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:38.432188 23401 consensus_meta.cc:287] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.786s user 0.006s sys 0.019s
I20251214 02:59:38.477766 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.479420 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.480810 21245 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.483004 21279 raft_consensus.cc:2393] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:38.482736 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:38.470790 23621 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
W20251214 02:59:38.471100 23597 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20251214 02:59:38.486861 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.487718 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:38.488623 22016 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:38.466632 23538 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.402796 23349 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.498108 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:38.492440 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 111905us
I20251214 02:59:38.454339 23504 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:38.497069 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:38.500296 21098 leader_election.cc:336] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): Timed out: RequestConsensusVote RPC to 127.19.131.67:34525 timed out after 1.749s (SENT)
I20251214 02:59:38.500721 21098 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.503182 21245 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.490867 23393 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:38.508410 21245 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.510303 21171 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:38.509151 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:38.514853 21207 raft_consensus.cc:1240] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:38.499375 23475 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:38.520452 23615 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:38.514613 23504 raft_consensus.cc:697] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:38.493191 23628 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.490628 23401 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.508034 23479 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.495335 23592 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20251214 02:59:38.509534 23613 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:38.526194 21245 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:38.514514 23545 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.512297 23622 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.512048 23626 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.433141 23604 consensus_queue.cc:260] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.539059 23631 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:38.512516 23550 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20251214 02:59:38.547300 21276 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
W20251214 02:59:38.556082 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 45862.3us
I20251214 02:59:38.535661 23515 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.545820 23628 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.563872 22011 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:38.565320 22015 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
W20251214 02:59:38.556529 23531 consensus_meta.cc:287] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.703s user 0.012s sys 0.014s
I20251214 02:59:38.526594 23615 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.551577 23613 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:38.554622 23632 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:38.610126 23632 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:38.610800 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.612722 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.610631 23632 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:38.610548 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.557785 23546 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.646787 23581 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.648478 23401 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.557163 23631 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:38.650619 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.642824 23586 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.643606 23504 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.613433 23557 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.645040 23613 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.549923 23625 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20251214 02:59:38.651868 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:38.600126 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 102541us
I20251214 02:59:38.648993 23581 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:38.732781 21247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.010s sys 0.018s
W20251214 02:59:38.739796 21247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s user 0.011s sys 0.018s
W20251214 02:59:38.740732 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 773289us
I20251214 02:59:38.740893 21280 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:38.735991 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:38.737761 21985 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:38.750888 21205 raft_consensus.cc:1275] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:38.684717 23577 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:38.761809 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 221521us
I20251214 02:59:38.763397 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:38.732125 23628 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:38.747838 23489 consensus_meta.cc:287] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.514s user 0.005s sys 0.005s
I20251214 02:59:38.573616 23531 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:38.768930 23399 consensus_meta.cc:287] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.554s user 0.006s sys 0.002s
I20251214 02:59:38.769340 23399 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.769942 23531 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.572695 23595 consensus_queue.cc:1059] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.164s
I20251214 02:59:38.771183 23595 raft_consensus.cc:3055] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:38.771557 23595 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:38.771744 23586 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.767092 23349 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.767378 23610 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:38.767834 23628 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.775512 21202 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:38.773115 23546 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:38.779623 21205 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:38.779917 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:38.575484 23602 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.200s
I20251214 02:59:38.783219 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:38.783463 23615 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.765791 23357 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:38.767120 23420 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:38.784407 23545 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:38.784989 23257 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:38.779179 23349 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:38.777148 22015 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251214 02:59:38.788818 23552 consensus_meta.cc:287] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.732s user 0.014s sys 0.000s
I20251214 02:59:38.741729 23631 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.759079 23550 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20251214 02:59:38.783268 23497 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:38.790827 23399 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.791453 21277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1812) took 1360 ms. Trace:
I20251214 02:59:38.792150 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:38.792848 23595 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.794282 21200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1260) took 1827 ms. Trace:
I20251214 02:59:38.874651 21200 rpcz_store.cc:276] 1214 02:59:36.967029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.126889 (+159860us) service_pool.cc:225] Handling call
1214 02:59:37.142062 (+ 15173us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.221248 (+1079186us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:38.794052 (+572804us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000e9054. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:38.794209 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2480778624}
I20251214 02:59:38.867405 23257 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.886372 21205 raft_consensus.cc:606] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:38.886819 21205 raft_consensus.cc:614] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:38.941367 23609 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.867623 23552 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:38.864688 23349 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:38.942777 23399 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:38.854475 23615 consensus_queue.cc:1048] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251214 02:59:38.945740 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:38.946359 22015 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:38.876611 23545 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:38.781575 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:38.797729 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:38.780745 23616 consensus_meta.cc:287] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.548s user 0.010s sys 0.004s
W20251214 02:59:38.810850 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 301220us
W20251214 02:59:38.812446 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.043074 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.043707 21203 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:38.789841 22016 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:38.789728 23628 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:38.855438 23420 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:38.862368 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 124711us
I20251214 02:59:38.867951 21277 rpcz_store.cc:276] 1214 02:59:37.431371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.437025 (+ 5654us) service_pool.cc:225] Handling call
1214 02:59:37.438462 (+ 1437us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.213317 (+774855us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:38.791244 (+577927us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54003cc8d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:38.791400 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1744006400}
W20251214 02:59:39.021998 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 240511us
W20251214 02:59:39.040488 23638 log.cc:927] Time spent T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20251214 02:59:39.041114 22013 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 02:59:38.944269 23349 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:38.958222 23257 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:39.114349 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 60754us
W20251214 02:59:39.115646 21199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2524) took 2471 ms (client timeout 2067 ms). Trace:
W20251214 02:59:39.116024 21199 rpcz_store.cc:269] 1214 02:59:36.644468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:36.644781 (+ 313us) service_pool.cc:225] Handling call
1214 02:59:39.115455 (+2470674us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54001bc1d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.115581 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":865411328}
W20251214 02:59:39.118108 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 76456.3us
I20251214 02:59:39.114948 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.051018 23595 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:39.134605 22016 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:39.139775 23641 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:39.120292 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.143354 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.153524 23420 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:39.154116 21279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 2542) took 1253 ms (client timeout 1635 ms). Trace:
W20251214 02:59:39.154548 21279 rpcz_store.cc:269] 1214 02:59:37.900847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.116208 (+215361us) service_pool.cc:225] Handling call
1214 02:59:39.153879 (+1037671us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540026a554. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.154050 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729783680}
I20251214 02:59:39.155756 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.156291 23497 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:39.156770 23497 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:39.157012 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.157630 21279 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:39.159001 21275 raft_consensus.cc:1240] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:39.158447 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.160415 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.160909 21279 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:39.161736 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.162720 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.163221 21279 raft_consensus.cc:2376] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10. Re-sending same reply.
I20251214 02:59:39.162448 23605 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:39.041189 23616 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.168445 23616 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:39.062060 23531 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20251214 02:59:39.172477 21277 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:39.174069 23573 log.cc:927] Time spent T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.547s user 0.016s sys 0.010s
I20251214 02:59:39.177819 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.209693 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.210956 21199 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:39.180171 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.180861 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.183207 22020 raft_consensus.cc:1240] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:39.213981 22020 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 568) took 1102 ms. Trace:
I20251214 02:59:39.214318 22020 rpcz_store.cc:276] 1214 02:59:38.111091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.111236 (+ 145us) service_pool.cc:225] Handling call
1214 02:59:39.182799 (+1071563us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b5800212a98. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:39.183021 (+ 222us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.213732 (+ 30711us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.213931 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18053888}
I20251214 02:59:39.129637 23626 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20251214 02:59:39.186497 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1801) took 2022 ms. Trace:
I20251214 02:59:39.219208 21278 rpcz_store.cc:276] 1214 02:59:37.163906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.164022 (+ 116us) service_pool.cc:225] Handling call
1214 02:59:37.164259 (+ 237us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.461170 (+1296911us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.186259 (+725089us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400002b54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.186439 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2981601152}
I20251214 02:59:39.163460 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:39.220403 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:39.207877 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 76320.4us
I20251214 02:59:39.187660 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:39.227896 22019 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44282 (request call id 569) took 1067 ms. Trace:
I20251214 02:59:39.228230 22019 rpcz_store.cc:276] 1214 02:59:38.160351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.163273 (+ 2922us) service_pool.cc:225] Handling call
1214 02:59:39.187500 (+1024227us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b5800212a98. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f53bd0 00007f6713f534b4 00007f670d4e3834 00007f670d4e37c2
1214 02:59:39.227812 (+ 40312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13672704}
I20251214 02:59:39.171391 23641 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.223644 21204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2543) took 2260 ms. Trace:
I20251214 02:59:39.235879 21204 rpcz_store.cc:276] 1214 02:59:36.963374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.093760 (+130386us) service_pool.cc:225] Handling call
1214 02:59:37.094051 (+ 291us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:37.094286 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:37.094309 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:37.094726 (+ 417us) log.cc:844] Serialized 26 byte log entry
1214 02:59:37.267504 (+172778us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:37.267547 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:37.267628 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:39.170375 (+1902747us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:39.223377 (+ 53002us) raft_consensus.cc:1685] finished
1214 02:59:39.223396 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:39.223587 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9389952,"thread_start_us":127804,"threads_started":1,"wal-append.queue_time_us":508486}
I20251214 02:59:39.224380 21203 raft_consensus.cc:1240] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:39.172089 23497 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:39.281975 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.282437 23600 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:39.282968 23600 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.283582 21278 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 02:59:39.284165 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2553) took 1219 ms. Trace:
I20251214 02:59:39.284452 21278 rpcz_store.cc:276] 1214 02:59:38.064422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:39.283021 (+1218599us) service_pool.cc:225] Handling call
1214 02:59:39.283291 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.283939 (+ 648us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.284119 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:39.288208 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.160805 23545 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.237197 23641 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:39.236234 23257 consensus_queue.cc:1059] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
W20251214 02:59:39.304312 23593 consensus_meta.cc:287] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.877s user 0.002s sys 0.008s
I20251214 02:59:39.237037 23628 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 5, Time since last communication: 0.065s
I20251214 02:59:39.290364 23609 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:39.305305 23504 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.032s sys 0.083s
I20251214 02:59:39.308362 23593 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.325477 23593 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:39.323378 23613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.058s sys 0.091s
W20251214 02:59:39.326470 23613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.058s sys 0.091s
W20251214 02:59:39.347275 23632 consensus_meta.cc:287] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.701s user 0.004s sys 0.005s
I20251214 02:59:39.347792 23632 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.349785 23632 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.360960 23641 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.363293 23641 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:39.391104 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 225470us
I20251214 02:59:39.392493 21208 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:39.395866 22014 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:39.401304 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:39.404078 23641 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:39.404956 21282 raft_consensus.cc:1240] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:39.406136 21200 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:39.406535 23531 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:39.407840 23531 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.409507 23609 consensus_queue.cc:765] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 5 is ahead of the local log (next sequential op: 5). Destination peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.173s
I20251214 02:59:39.411867 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.412945 23531 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:39.413735 23531 consensus_queue.cc:1059] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:39.415167 23531 raft_consensus.cc:3055] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:39.415611 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.415594 23531 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.393854 21171 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.423877 23529 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 02:59:39.332610 23504 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.032s sys 0.083s
I20251214 02:59:39.435119 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.435941 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 281) took 2274 ms. Trace:
I20251214 02:59:39.436141 21204 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.436965 21204 raft_consensus.cc:2410] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251214 02:59:39.436476 21201 rpcz_store.cc:276] 1214 02:59:37.161723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.170456 (+ 8733us) service_pool.cc:225] Handling call
1214 02:59:37.242719 (+ 72263us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:37.636537 (+393818us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:37.636564 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:37.636604 (+ 40us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:37.636634 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.435336 (+1798702us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540008f054. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.435431 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:39.435689 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":905274752}
I20251214 02:59:39.439028 21273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1813) took 1997 ms. Trace:
I20251214 02:59:39.439424 21273 rpcz_store.cc:276] 1214 02:59:37.441672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.441812 (+ 140us) service_pool.cc:225] Handling call
1214 02:59:37.442146 (+ 334us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.367247 (+925101us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.438520 (+1071273us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54002fd5d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.438831 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2053571840}
I20251214 02:59:39.440233 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1269) took 1068 ms. Trace:
I20251214 02:59:39.440659 21208 rpcz_store.cc:276] 1214 02:59:38.371333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.371518 (+ 185us) service_pool.cc:225] Handling call
1214 02:59:39.115611 (+744093us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.393447 (+277836us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.439189 (+ 45742us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54001bc1d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.439940 (+ 751us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54001bc1d0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.440136 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2292347520}
I20251214 02:59:39.438482 21199 raft_consensus.cc:1240] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:39.436977 21202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47944 (request call id 288) took 1198 ms. Trace:
I20251214 02:59:39.443073 21202 rpcz_store.cc:276] 1214 02:59:38.238466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.268648 (+ 30182us) service_pool.cc:225] Handling call
1214 02:59:39.436549 (+1167901us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b540037afd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.436801 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":925375360}
W20251214 02:59:39.442752 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 299352us
I20251214 02:59:39.442852 23497 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:39.447870 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 47001.5us
I20251214 02:59:39.449529 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:39.450017 23497 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:39.450950 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 396751us
I20251214 02:59:39.459555 22017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44274 (request call id 507) took 1425 ms. Trace:
I20251214 02:59:39.460009 22017 rpcz_store.cc:276] 1214 02:59:38.034084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.034284 (+ 200us) service_pool.cc:225] Handling call
1214 02:59:39.457801 (+1423517us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b5800212a98. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 02:59:39.458221 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.458771 (+ 550us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:39.458811 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:39.458903 (+ 92us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:39.458947 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.459038 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:39.459206 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4313472}
I20251214 02:59:39.454610 23628 consensus_queue.cc:1059] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251214 02:59:39.457510 21281 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1799) took 2294 ms. Trace:
I20251214 02:59:39.461851 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.462086 23628 raft_consensus.cc:3055] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:39.462581 23628 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.463977 23628 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.465747 23628 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.467075 23257 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 02:59:39.464001 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 70707.8us
I20251214 02:59:39.461407 21281 rpcz_store.cc:276] 1214 02:59:37.163240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.163370 (+ 130us) service_pool.cc:225] Handling call
1214 02:59:37.163652 (+ 282us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.154177 (+990525us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.456882 (+1302705us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54003cd2d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.457292 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2164987136}
I20251214 02:59:39.474109 22017 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:39.479513 23593 consensus_queue.cc:1059] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:39.480930 23593 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:39.480813 21985 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.482532 23592 raft_consensus.cc:3055] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:39.484977 23592 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.486385 23592 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.488399 23592 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.445071 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.483698 23456 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:39.479480 21279 raft_consensus.cc:3055] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:39.495314 21279 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.495743 21280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1822) took 1587 ms. Trace:
I20251214 02:59:39.496201 21280 rpcz_store.cc:276] 1214 02:59:37.908431 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.455307 (+546876us) service_pool.cc:225] Handling call
1214 02:59:38.483904 (+ 28597us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.741287 (+257383us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.495200 (+753913us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54001fd6d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.495502 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":590652672}
I20251214 02:59:39.496469 21279 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.497859 21279 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:39.428910 23531 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.521344 21274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 292) took 1613 ms. Trace:
I20251214 02:59:39.521961 22011 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
W20251214 02:59:39.523464 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 74233.6us
I20251214 02:59:39.524542 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.526160 23402 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:39.521813 21274 rpcz_store.cc:276] 1214 02:59:37.907644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.402412 (+494768us) service_pool.cc:225] Handling call
1214 02:59:38.403945 (+ 1533us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.436449 (+ 32504us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:38.436483 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:38.436525 (+ 42us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:38.436556 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.520882 (+1084326us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b54001d4a54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.520948 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:39.521160 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74257792}
I20251214 02:59:39.526886 23471 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:39.527627 23471 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.528577 22015 raft_consensus.cc:1240] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:39.529500 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:39.531167 23545 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:39.531755 23545 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.531939 22018 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 02:59:39.532574 23545 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:39.530313 22020 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:39.481159 23257 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20251214 02:59:39.531275 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:39.536688 22016 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.537279 22016 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
W20251214 02:59:39.538678 23292 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.000s sys 0.016s
W20251214 02:59:39.539144 23292 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.000s sys 0.017s
I20251214 02:59:39.540225 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.544596 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.545398 23217 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:39.546283 23461 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:39.547245 23609 consensus_queue.cc:1059] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:39.548499 23615 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.549253 23615 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.549748 23609 raft_consensus.cc:3055] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:39.550187 23609 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.550508 23402 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.535825 23407 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:39.554059 23615 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.555100 23600 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:39.572283 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49657.4us
I20251214 02:59:39.578219 21245 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.565431 23471 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:39.587678 23402 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:39.588829 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:39.595880 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:39.594255 23461 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 02:59:39.604039 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 62028.2us
I20251214 02:59:39.609432 21279 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:39.614816 22019 raft_consensus.cc:1240] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:39.617090 21173 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.617246 23557 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251214 02:59:39.618170 23557 consensus_queue.cc:1243] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer 668ad4f30bd649c88a034bbab9f13b20 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20251214 02:59:39.618633 23611 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:39.619825 22016 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20251214 02:59:39.620795 23624 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.144s user 0.008s sys 0.013s
I20251214 02:59:39.623023 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2563) took 1416 ms. Trace:
I20251214 02:59:39.623281 21127 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:39.623570 23586 raft_consensus.cc:3055] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:39.624044 23586 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.625381 23586 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.626915 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:39.627818 21277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 2559) took 1158 ms (client timeout 1539 ms). Trace:
W20251214 02:59:39.628297 21277 rpcz_store.cc:269] 1214 02:59:38.469015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:39.047412 (+578397us) service_pool.cc:225] Handling call
1214 02:59:39.627443 (+580031us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b54001d4554. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.627642 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117710208}
I20251214 02:59:39.631634 23604 raft_consensus.cc:3060] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:39.629185 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:39.614379 23617 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.021s sys 0.017s
I20251214 02:59:39.632123 21208 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:39.635200 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.623347 21206 rpcz_store.cc:276] 1214 02:59:38.206652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.206798 (+ 146us) service_pool.cc:225] Handling call
1214 02:59:38.214927 (+ 8129us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:38.216102 (+ 1175us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:38.216130 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:38.216551 (+ 421us) log.cc:844] Serialized 26 byte log entry
1214 02:59:38.368414 (+151863us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:38.368450 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:38.368511 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:39.619904 (+1251393us) spinlock_profiling.cc:243] Waited 7.26 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:39.622542 (+ 2638us) raft_consensus.cc:1685] finished
1214 02:59:39.622581 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:39.622850 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":801,"raft.queue_time_us":1518,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":16699776,"thread_start_us":27599,"threads_started":1,"wal-append.queue_time_us":214936}
W20251214 02:59:39.637404 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 75098.5us
I20251214 02:59:39.637964 23609 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.639250 21245 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.639154 23611 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:39.640010 23593 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:39.640784 23221 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:39.640273 23611 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.650251 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:39.650806 23592 consensus_queue.cc:1059] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:39.649014 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.652045 23557 raft_consensus.cc:3055] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:39.652104 23612 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:39.652561 23557 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:39.653035 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.653225 23622 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:39.653666 23622 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.654009 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.653702 23557 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.654891 23622 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.656553 23622 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:39.657111 23596 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:39.658115 23612 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:39.655431 23557 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:39.636101 23617 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.165s user 0.022s sys 0.017s
I20251214 02:59:39.660425 23596 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:39.663506 23584 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:39.664037 21276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1804) took 2499 ms. Trace:
I20251214 02:59:39.665004 21276 rpcz_store.cc:276] 1214 02:59:37.164725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:37.164990 (+ 265us) service_pool.cc:225] Handling call
1214 02:59:37.165274 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:38.554351 (+1389077us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.663577 (+1109226us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540005bc54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.663872 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3178149760}
I20251214 02:59:39.639436 23609 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.669377 23531 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:39.671981 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 68885.5us
I20251214 02:59:39.673671 23552 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:39.658701 23612 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:39.662091 23590 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.588s user 0.019s sys 0.023s
W20251214 02:59:39.696135 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 46075.1us
W20251214 02:59:39.695693 23590 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.637s user 0.019s sys 0.023s
I20251214 02:59:39.708762 23586 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:39.716616 21130 raft_consensus.cc:1240] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:39.718822 23652 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20251214 02:59:39.721331 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.722015 23457 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:39.723546 21202 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b98a871e4a86480da87fed8511a68575"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:39.724334 21202 raft_consensus.cc:606] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:39.735677 23545 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.737803 23545 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.746968 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:39.747620 23551 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:39.748152 23551 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:39.749366 23551 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.756487 23600 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:39.768258 21171 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:39.740772 23587 consensus_queue.cc:1059] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:39.750512 23296 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:39.796347 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.798754 22014 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 02:59:39.803362 23603 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:39.803874 23603 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.805038 23603 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:39.805718 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:39.806257 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.806461 23653 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:39.806756 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.803403 21096 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.807852 23560 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
W20251214 02:59:39.808562 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 178115us
I20251214 02:59:39.809063 23533 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:39.809669 23533 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.810899 23214 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:39.813537 23112 consensus_queue.cc:1059] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:39.813640 23635 consensus_queue.cc:1059] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:39.814908 23112 raft_consensus.cc:3055] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:39.815102 23635 raft_consensus.cc:3055] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:39.815333 23112 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.815667 23560 raft_consensus.cc:697] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.816851 23560 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.818373 23635 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.809507 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.821734 21275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50516 (request call id 1821) took 1913 ms. Trace:
W20251214 02:59:39.822811 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.822297 21275 rpcz_store.cc:276] 1214 02:59:37.908287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.452719 (+544432us) service_pool.cc:225] Handling call
1214 02:59:39.158795 (+706076us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.159367 (+ 572us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.821367 (+662000us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b540026a550. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:39.821565 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1572142848}
I20251214 02:59:39.829137 21099 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:39.829821 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 162766us
I20251214 02:59:39.830731 23653 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.833035 23653 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:39.835191 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:39.843406 23603 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:39.844444 23640 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.830444 23292 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:39.848990 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
W20251214 02:59:39.851182 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 158259us
I20251214 02:59:39.855932 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.850229 21199 raft_consensus.cc:606] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:39.858867 21203 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:39.864135 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:39.837417 23296 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20251214 02:59:39.870442 21282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 288) took 1965 ms. Trace:
I20251214 02:59:39.870816 23577 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:39.871407 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 62064.3us
I20251214 02:59:39.871109 23112 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.872098 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.872517 23112 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:39.872753 21275 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 8.
I20251214 02:59:39.873601 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.858723 23629 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:39.870882 21282 rpcz_store.cc:276] 1214 02:59:37.905275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:38.297250 (+391975us) service_pool.cc:225] Handling call
1214 02:59:38.297535 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.405545 (+1108010us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:39.867279 (+461734us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003890d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:39.870379 (+ 3100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2545766784}
I20251214 02:59:39.890977 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20251214 02:59:39.893482 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.881603 21985 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:39.862888 23584 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.821660 23635 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.881158 23292 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:39.857307 23640 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.870857 23533 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.880307 23622 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:39.892714 23636 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20251214 02:59:39.894829 23604 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:39.912942 21199 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 02:59:39.906383 23292 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:39.914327 23583 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.914247 23543 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:39.918476 23584 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.918180 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:39.919442 23533 consensus_queue.cc:1059] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:39.927356 23533 raft_consensus.cc:3055] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:39.927709 23533 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:39.927978 23640 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.928776 23533 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.930059 23533 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:39.916034 23577 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.918270 23585 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:39.932746 23583 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.935261 23635 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.923210 23292 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:39.940562 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.908608 23340 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20251214 02:59:39.942389 23543 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:39.937201 23622 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.949309 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.949331 23340 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:39.950035 23578 raft_consensus.cc:3055] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:39.942979 23546 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:39.944226 23584 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.944588 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:39.941252 23604 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:39.964835 23604 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:39.966922 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.967324 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.967561 21275 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:39.967936 21276 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:39.968343 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:39.968290 23112 consensus_queue.cc:1059] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:39.971155 23240 consensus_queue.cc:1059] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:39.971712 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.972415 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:39.972647 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.972795 23240 raft_consensus.cc:3055] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:39.973188 23240 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.973935 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:39.974603 21129 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:39.975086 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:39.975807 23112 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:39.976209 23112 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.971216 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:39.979552 23112 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:39.982517 23112 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
W20251214 02:59:39.982682 21248 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:39.948285 23583 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:39.946027 23577 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:39.950392 21206 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:39.987946 23622 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:39.950793 23340 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:39.954751 21278 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:39.954733 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.955309 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.951625 23543 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:39.950453 23578 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:39.961989 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:39.962604 23629 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:39.974161 23240 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.000587 23240 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:39.976403 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:39.939437 23611 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:39.996295 21129 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:39.949668 23601 raft_consensus.cc:3055] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:40.004534 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:39.998082 21248 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:40.020673 21132 raft_consensus.cc:2393] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:40.021467 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2555) took 1553 ms. Trace:
I20251214 02:59:40.003957 23601 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:40.023700 23601 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:39.998863 23629 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:40.025379 23629 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:40.026441 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.027738 21124 raft_consensus.cc:1240] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:40.028244 23402 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:40.028800 23402 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.030141 21276 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251214 02:59:40.030921 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 69057.2us
I20251214 02:59:40.031405 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:40.031687 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:40.032933 23471 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:40.033492 23471 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:40.033663 23551 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.021785 21278 rpcz_store.cc:276] 1214 02:59:38.468152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:39.285119 (+816967us) service_pool.cc:225] Handling call
1214 02:59:39.286727 (+ 1608us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:39.998743 (+712016us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:40.001536 (+ 2793us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b540026b954. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:40.021405 (+ 19869us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535138176}
I20251214 02:59:39.997328 23543 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.036355 21204 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:40.039803 21275 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:40.042106 23489 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20251214 02:59:40.042695 23471 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:40.043531 23676 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20251214 02:59:40.044123 23445 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:40.035123 23402 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.025010 23601 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.045969 21099 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:40.035517 23596 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251214 02:59:40.086246 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.086736 21205 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:40.086375 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 83263.9us
I20251214 02:59:40.000684 23578 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.041347 23604 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:40.092171 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 46558.2us
I20251214 02:59:40.095794 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:40.108623 21986 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.110036 23340 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.117889 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:40.118621 22015 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:40.117913 23670 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:40.097553 23402 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:40.121979 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.127281 23611 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:40.128187 23640 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.127997 23611 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.134250 23611 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.134310 23445 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:40.134841 23445 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:40.129372 21129 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:40.121235 23670 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:40.140542 21277 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:40.099332 23604 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.125337 23402 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:40.151497 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.143409 23670 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.162284 21277 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.152902 23402 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:40.140026 23217 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251214 02:59:40.167668 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:40.169389 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.171245 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.171638 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.149195 23612 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.160562 23578 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
W20251214 02:59:40.176246 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.168694 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:40.178220 23679 log.cc:927] Time spent T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20251214 02:59:40.179021 21275 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.181088 21127 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
W20251214 02:59:40.181444 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.183127 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.186863 21202 raft_consensus.cc:1240] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:40.188176 23129 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:40.188747 23129 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.189442 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.190233 21280 raft_consensus.cc:2393] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:40.192749 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.191422 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:40.199246 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.199839 21273 raft_consensus.cc:1813] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Handling vote request from an unknown peer 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:40.200371 21273 raft_consensus.cc:2410] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20251214 02:59:40.201112 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.195996 23340 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:40.206895 23129 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.216158 21276 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
W20251214 02:59:40.223845 23557 consensus_meta.cc:287] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.549s user 0.005s sys 0.005s
I20251214 02:59:40.225760 23612 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.230552 21279 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 02:59:40.237135 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.240674 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.233799 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.243984 21276 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:40.244920 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.254263 21127 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:40.244052 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:40.235637 22016 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:40.258960 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.263587 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.269408 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:40.270967 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:40.271302 23360 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:40.271746 23360 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:40.271906 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.272760 23357 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:40.273269 23357 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.275161 23357 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.275683 23357 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:40.276106 23357 raft_consensus.cc:697] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.277006 23357 consensus_queue.cc:237] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:40.278664 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.283188 23360 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:40.285574 21276 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.289381 23667 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:40.295070 21275 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:40.239470 23629 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.260346 23640 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:40.299139 21125 raft_consensus.cc:1240] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:40.262805 23604 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:40.304201 21127 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
W20251214 02:59:40.298669 21247 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.504s (SENT) [suppressed 30 similar messages]
I20251214 02:59:40.258381 23445 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.309531 21133 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.272292 23349 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20251214 02:59:40.312408 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.327739 21276 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:40.335078 22015 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:40.336652 21275 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 02:59:40.340479 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.332664 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.345487 21273 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:40.343304 21171 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:40.355494 21127 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:40.357311 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.353065 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.358955 21276 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.362015 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.364493 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.365171 21276 raft_consensus.cc:2393] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.367108 21275 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.370896 21273 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:40.367571 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 96706us
I20251214 02:59:40.373044 21276 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:40.376128 23402 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.358146 23349 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:40.377977 21282 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 02:59:40.217403 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:40.379233 21276 raft_consensus.cc:1240] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:40.382737 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.377969 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 79328.8us
I20251214 02:59:40.389603 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.392912 21273 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:40.393262 21275 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:40.395135 21276 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 02:59:40.396056 21132 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:40.389394 23221 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:40.416682 23221 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.417788 23221 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.418870 23221 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:40.392997 23349 raft_consensus.cc:697] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.423867 23349 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.392550 23356 consensus_queue.cc:1059] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20251214 02:59:40.402637 21125 raft_consensus.cc:1240] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:40.404598 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:40.406143 23402 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.409422 21276 raft_consensus.cc:1240] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
W20251214 02:59:40.409705 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.414997 22015 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:40.416170 23445 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:40.401002 23685 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20251214 02:59:40.434264 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.435552 21276 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.436854 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:40.440737 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:40.449102 21173 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:40.452360 21202 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:40.450388 23560 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:40.455389 21274 raft_consensus.cc:2410] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251214 02:59:40.456321 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.460130 21276 raft_consensus.cc:1275] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 02:59:40.458629 21099 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:40.461855 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.462361 21275 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:40.406198 23543 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.467638 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
W20251214 02:59:40.470993 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.469604 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.450312 23670 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.457217 23629 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.472653 21276 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.475395 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.455014 23690 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20251214 02:59:40.466322 23217 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:40.477016 21124 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.481416 21280 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:40.471773 23360 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.484769 21275 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 02:59:40.487635 21985 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:40.488536 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.490672 21275 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:40.485323 23217 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 02:59:40.489707 23635 consensus_meta.cc:287] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.000s
I20251214 02:59:40.499257 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.500977 23217 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.499614 21207 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:40.494326 23445 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election won for term 9
W20251214 02:59:40.496902 23292 consensus_meta.cc:287] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.504s user 0.009s sys 0.001s
I20251214 02:59:40.501046 23612 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20251214 02:59:40.535228 23612 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:40.535591 23612 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.536417 23612 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.518651 23670 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:40.521565 23360 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.523818 23445 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:40.529065 21125 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.544020 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:40.529663 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.529976 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.544821 21131 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:40.530102 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.530953 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.530999 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.530949 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.533247 23595 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:40.538576 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.537658 23612 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:40.550340 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.551343 21125 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.552930 21281 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20251214 02:59:40.552300 21130 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:40.516144 23551 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:40.554790 21128 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:40.553262 21281 pending_rounds.cc:85] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 7
I20251214 02:59:40.516439 23292 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.556296 21281 pending_rounds.cc:107] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20251214 02:59:40.558669 21281 raft_consensus.cc:2889] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:40.559737 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.547389 23595 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:40.573346 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:40.573426 21208 raft_consensus.cc:1240] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:40.578378 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:40.585783 23545 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:40.586408 23545 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.546057 23340 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20251214 02:59:40.588321 23545 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.589583 23667 consensus_queue.cc:1059] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:40.590612 23667 raft_consensus.cc:3055] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:40.590945 23667 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:40.591674 23667 consensus_queue.cc:260] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.592741 23667 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:40.593596 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.594800 23445 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:40.598095 23688 log.cc:927] Time spent T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.196s user 0.002s sys 0.001s
I20251214 02:59:40.601052 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:40.604130 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45666.7us
I20251214 02:59:40.611490 21131 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:40.616490 21125 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:40.562733 23690 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20251214 02:59:40.573063 23543 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:40.619963 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.653542 21279 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 10.
I20251214 02:59:40.660307 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 156) took 9652 ms. Trace:
I20251214 02:59:40.603705 23551 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.665001 23551 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.663923 23585 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Committing config change with OpId 5.7: config changed from index 4 to 7, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:40.666007 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.667141 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.668149 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.669132 21278 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:40.625741 21276 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:40.626374 22014 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9. Ops: []
I20251214 02:59:40.674304 21129 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
W20251214 02:59:40.675274 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.674911 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.676518 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.677105 21279 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.677520 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.678043 21131 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.680897 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:40.682150 23653 raft_consensus.cc:2764] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251214 02:59:40.684156 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.684232 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:40.684942 23560 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:40.685393 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.606141 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.690171 23296 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20251214 02:59:40.690791 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 48294.2us
I20251214 02:59:40.690843 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.630781 23340 raft_consensus.cc:3055] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:40.631291 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.685431 21208 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 5
I20251214 02:59:40.692260 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.632160 23690 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:40.633858 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:40.692893 23690 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.634811 23129 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:40.693269 23548 consensus_queue.cc:1059] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 02:59:40.634843 21282 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:40.636060 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.678470 21131 raft_consensus.cc:2368] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:40.691007 23296 consensus_queue.cc:1243] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 8de0cf3ca4bf4083bc494a9af0e49342 log is divergent from this leader: its last log entry 9.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20251214 02:59:40.694453 23360 consensus_queue.cc:1059] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:40.694774 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.643931 23595 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.695940 23340 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.696442 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.696624 23595 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.693969 23546 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20251214 02:59:40.698511 23546 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:40.675331 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.693821 23690 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.700402 23690 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.698824 23546 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.702528 23546 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.703936 23450 raft_consensus.cc:3055] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:40.704300 23450 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.705119 23450 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.695701 23360 raft_consensus.cc:3055] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:40.707218 23360 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.649995 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:40.707751 22019 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:40.650923 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.652638 23636 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:40.671684 21029 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index 4 to 3, term changed from 2 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from VOTER to NON_VOTER, now has a pending config: VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65), VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66), VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67), VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:40.682030 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.693893 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:40.692512 23112 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:40.693809 23129 raft_consensus.cc:697] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.695313 23635 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:40.660712 21027 rpcz_store.cc:276] 1214 02:59:31.008037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:31.008381 (+ 344us) service_pool.cc:225] Handling call
1214 02:59:40.660079 (+9651698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:37.624994 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:37.648284 (+ 23290us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:37.648322 (+ 38us) write_op.cc:435] Acquired schema lock
1214 02:59:37.648339 (+ 17us) tablet.cc:660] Decoding operations
1214 02:59:37.649097 (+ 758us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:37.649158 (+ 61us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:37.649184 (+ 26us) tablet.cc:683] Acquiring locks for 7 operations
1214 02:59:37.649703 (+ 519us) tablet.cc:699] Row locks acquired
1214 02:59:37.649717 (+ 14us) write_op.cc:260] PREPARE: finished
1214 02:59:37.649855 (+ 138us) write_op.cc:270] Start()
1214 02:59:37.649958 (+ 103us) write_op.cc:276] Timestamp: P: 1765681177649807 usec, L: 0
1214 02:59:37.649975 (+ 17us) op_driver.cc:348] REPLICATION: starting
1214 02:59:37.787430 (+137455us) log.cc:844] Serialized 2831 byte log entry
1214 02:59:38.470403 (+682973us) op_driver.cc:464] REPLICATION: finished
1214 02:59:39.452683 (+982280us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f67129d6e40
1214 02:59:39.694693 (+242010us) write_op.cc:301] APPLY: starting
1214 02:59:39.704882 (+ 10189us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:39.708777 (+ 3895us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:39.708812 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:39.716909 (+ 8097us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:39.717209 (+ 300us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1214 02:59:39.717250 (+ 41us) write_op.cc:312] APPLY: finished
1214 02:59:39.717629 (+ 379us) log.cc:844] Serialized 59 byte log entry
1214 02:59:39.852628 (+134999us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:39.896712 (+ 44084us) write_op.cc:454] Released schema lock
1214 02:59:39.904902 (+ 8190us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:39.943740 (+ 38838us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:39.943775 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1702467,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":1178095,"apply.run_cpu_time_us":97888,"apply.run_wall_time_us":279853,"num_ops":7,"prepare.queue_time_us":393314,"prepare.run_cpu_time_us":27613,"prepare.run_wall_time_us":393149,"raft.queue_time_us":280587,"raft.run_cpu_time_us":40519,"raft.run_wall_time_us":1265835,"replication_time_us":785702,"spinlock_wait_cycles":6693248,"thread_start_us":1859438,"threads_started":5,"wal-append.queue_time_us":490783,"wal-append.run_cpu_time_us":36879,"wal-append.run_wall_time_us":1312112}]]}
I20251214 02:59:40.697418 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.648922 23292 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:40.699936 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.697502 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:40.629808 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.729705 21208 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:40.729823 21131 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:40.640990 21130 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:40.718533 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.732050 21276 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:40.701874 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:40.732282 21201 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20251214 02:59:40.725682 21127 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:40.726316 21273 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:40.738772 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:40.743098 22015 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:40.745616 21274 raft_consensus.cc:1240] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:40.734086 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.709760 23450 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.749643 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:40.752794 21124 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:40.752785 23545 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20251214 02:59:40.744971 23635 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20251214 02:59:40.750447 23667 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:40.760318 23667 raft_consensus.cc:697] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:40.761231 23667 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.720198 23636 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.733230 23546 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.741441 23221 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:40.770925 21029 catalog_manager.cc:5654] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:40.719878 23603 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20251214 02:59:40.787225 21126 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:40.770267 23635 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:40.763365 23129 consensus_queue.cc:237] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.789409 21029 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:40.806097 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.815060 21133 raft_consensus.cc:1240] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:40.816591 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:40.820178 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.824397 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.827529 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 77977.5us
I20251214 02:59:40.828668 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:40.825831 21248 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:40.838054 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:40.839068 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:40.845892 23357 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 02:59:40.846443 23357 raft_consensus.cc:697] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:40.847534 23357 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:40.851039 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:40.854777 23349 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:40.854970 23221 consensus_queue.cc:1059] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:40.862100 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.854708 23635 consensus_queue.cc:1059] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20251214 02:59:40.856261 23612 consensus_queue.cc:1059] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20251214 02:59:40.867323 23340 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.867561 23629 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:40.868750 23340 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:40.867547 21098 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.874749 23402 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:40.876128 21098 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.876904 23402 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:40.883106 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:40.810214 23360 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.836081 23551 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:40.889904 23445 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:40.891970 22019 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:40.833750 23690 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20251214 02:59:40.939944 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:40.941509 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:40.824353 23636 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:40.956585 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:40.972970 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:40.995129 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 140782us
I20251214 02:59:40.952514 23292 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:40.996470 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:40.958498 23497 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:41.030858 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:40.940290 23445 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.031494 21132 raft_consensus.cc:2376] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6. Re-sending same reply.
I20251214 02:59:40.968783 23296 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20251214 02:59:41.040082 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.027035 23551 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.033583 23217 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:41.036790 23497 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:41.040134 23629 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:41.045461 23129 raft_consensus.cc:3055] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:41.065382 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.076947 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:41.081403 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:41.060707 23701 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20251214 02:59:41.054438 23217 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:41.058809 23129 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:41.049086 23551 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.126699 23551 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:41.128530 23550 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251214 02:59:41.131402 23706 log.cc:927] Time spent T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20251214 02:59:41.099071 23129 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:41.069885 23602 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 02:59:41.098462 23497 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:41.112459 23445 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:41.123170 23402 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:41.124190 23217 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:41.143361 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 146959us
I20251214 02:59:41.150550 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:41.151437 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:41.152109 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 195621us
W20251214 02:59:41.156265 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 115344us
I20251214 02:59:41.140662 23546 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:41.159946 21098 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:41.164464 21098 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:41.164592 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.165298 21202 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:41.177664 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.183218 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.178874 23129 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:41.191152 21279 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20251214 02:59:41.180120 23700 log.cc:927] Time spent T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.361s user 0.000s sys 0.003s
I20251214 02:59:41.192857 21245 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:41.193209 21986 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:41.169486 23545 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:41.176879 23640 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
W20251214 02:59:41.204860 23699 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
I20251214 02:59:41.205583 23722 consensus_queue.cc:1059] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:41.206426 23533 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:41.208024 23725 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:41.209551 22019 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:41.200557 23582 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:41.216478 23582 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:41.216809 23582 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:41.194000 23550 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:41.200855 23551 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:41.227921 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 60694.1us
I20251214 02:59:41.186729 23584 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20251214 02:59:41.241197 23705 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20251214 02:59:41.241365 23292 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:41.241662 23214 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Stepping down as leader of term 8
W20251214 02:59:41.242383 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 96965.5us
I20251214 02:59:41.243024 23357 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:41.194123 23667 consensus_queue.cc:1048] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251214 02:59:41.265317 21274 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
W20251214 02:59:41.268450 23704 log.cc:927] Time spent T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20251214 02:59:41.269325 23720 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
W20251214 02:59:41.272156 23710 log.cc:927] Time spent T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20251214 02:59:41.278180 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:41.285302 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 80091.1us
W20251214 02:59:41.295185 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.259277 23214 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:41.303294 21126 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:41.327327 22018 raft_consensus.cc:1217] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Deduplicated request from leader. Original: 7.8->[8.9-8.9] Dedup: 8.9->[]
I20251214 02:59:41.327885 22018 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20251214 02:59:41.313745 23713 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:41.333389 21132 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20251214 02:59:41.338637 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45751.4us
I20251214 02:59:41.339584 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.339697 23214 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.341289 21247 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.342072 23734 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:41.342825 23734 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:41.343493 23713 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.344671 23356 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.345148 23356 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.345592 23713 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:41.342486 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.352643 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 164245us
I20251214 02:59:41.348637 23690 raft_consensus.cc:3055] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
W20251214 02:59:41.373662 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.377197 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:41.353313 23670 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 02:59:41.382294 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 85522.9us
I20251214 02:59:41.384639 21098 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:41.385861 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:41.373291 23690 raft_consensus.cc:740] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:41.396878 21276 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20251214 02:59:41.398173 21247 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.393194 23733 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:41.371708 23720 consensus_queue.cc:1059] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20251214 02:59:41.409202 21247 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.410099 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 61557.3us
I20251214 02:59:41.380750 23722 consensus_queue.cc:1059] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20251214 02:59:41.393914 23667 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:41.391829 23731 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:41.435256 23551 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251214 02:59:41.440356 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 56273.9us
I20251214 02:59:41.439297 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.426079 23550 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:41.454823 23741 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.455436 23741 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.457538 23741 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:41.460619 21273 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:41.460567 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.442299 21274 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:41.461354 21202 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:41.447302 23356 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:41.448680 21247 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.453497 23450 consensus_meta.cc:287] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.685s user 0.010s sys 0.001s
I20251214 02:59:41.453315 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.465258 21132 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:41.462682 21986 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:41.467204 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:41.468412 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:41.468758 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:41.469054 21127 raft_consensus.cc:2376] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4. Re-sending same reply.
I20251214 02:59:41.464422 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.469771 21277 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:41.463693 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.470755 21279 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:41.435819 23713 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251214 02:59:41.461346 23640 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:41.473651 23582 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.474808 23551 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:41.475099 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:41.475353 23356 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:41.475755 23356 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:41.476078 23356 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:41.477891 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:41.478127 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:41.478500 21208 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:41.479729 23667 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:41.480239 23667 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:41.474381 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:41.481678 23582 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:41.482491 21124 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:41.483203 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:41.487078 22018 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 02:59:41.483914 23713 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20251214 02:59:41.494437 23731 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:41.496506 23731 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:41.496536 23746 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20251214 02:59:41.501753 23667 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:41.511377 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:41.508224 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 59636us
I20251214 02:59:41.520649 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:41.511317 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.513909 21275 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:41.510821 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.546340 21276 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:41.529115 21202 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 02:59:41.527248 21282 raft_consensus.cc:2393] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:41.553614 21247 leader_election.cc:400] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5.
I20251214 02:59:41.554608 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:41.530139 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.555905 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.557909 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.560911 21125 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:41.511245 23743 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.562383 21208 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 02:59:41.527055 23636 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:41.734769 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 171235us
I20251214 02:59:41.507927 23450 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:41.728084 21201 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:41.498550 23731 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:41.552631 23578 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:41.511452 23129 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20251214 02:59:41.754038 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.558473 23670 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:41.758558 23734 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.531428 23667 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.225s
I20251214 02:59:41.727111 23550 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:41.559451 23746 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.203s
I20251214 02:59:41.754045 23578 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:41.754485 23450 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:41.760419 23720 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.786522 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:41.788695 21201 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:41.790961 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:41.792021 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.727339 23743 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:41.811084 23743 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:41.812053 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 256196us
I20251214 02:59:41.812839 21247 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.814121 21247 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.788770 23578 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:41.789327 23450 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:41.827448 21274 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251214 02:59:41.759169 23734 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.834836 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:41.840554 21274 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:41.832721 23670 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:41.822815 23745 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:41.859500 23356 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:41.863246 23703 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.863s user 0.000s sys 0.002s
I20251214 02:59:41.868137 23746 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:41.868587 23757 consensus_queue.cc:1059] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:41.872892 23221 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:41.875598 23745 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:41.796564 23720 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:41.906113 23720 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:41.900110 23550 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:41.878654 21276 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:41.853886 21280 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:41.882737 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.909000 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57101.8us
I20251214 02:59:41.909704 21247 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:41.910742 23754 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:41.912662 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:41.898142 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:41.913792 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:41.901558 22019 raft_consensus.cc:2368] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
I20251214 02:59:41.904247 23734 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:41.899619 21274 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251214 02:59:41.855916 23743 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20251214 02:59:41.910995 23670 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:41.957551 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:42.021402 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:42.021298 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 108090us
W20251214 02:59:42.023291 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:42.023631 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:42.026160 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 108313us
I20251214 02:59:41.900206 23221 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:42.082386 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.082577 21280 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:41.956212 23734 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:42.115145 21279 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.000s sys 0.016s
I20251214 02:59:42.087949 23712 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:42.122694 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.089015 23720 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.125805 21125 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:42.126133 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.120065 21986 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:41.900569 23746 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:42.127568 21279 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.567s user 0.000s sys 0.017s
I20251214 02:59:42.130357 21130 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:42.115036 23754 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:42.134523 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:42.134755 23670 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.135281 23670 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.137190 23670 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:42.139075 23709 log.cc:927] Time spent T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.041s user 0.002s sys 0.000s
I20251214 02:59:42.113979 23760 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:42.140513 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 227919us
I20251214 02:59:42.141269 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:42.142383 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:42.144838 21127 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.000s sys 0.019s
W20251214 02:59:42.145154 21127 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.000s sys 0.019s
I20251214 02:59:42.149601 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:42.138810 23221 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.125591 23712 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.114821 23548 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20251214 02:59:42.154528 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 309789us
I20251214 02:59:42.120005 23734 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.152516 23670 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:42.182281 23743 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.182837 23743 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.183475 23723 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.183939 23723 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.184803 23743 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.185842 23723 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.187266 23731 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20251214 02:59:42.187661 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.188272 21125 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:42.192875 21171 leader_election.cc:400] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
I20251214 02:59:42.192971 23756 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:42.193609 23756 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:42.188095 23721 raft_consensus.cc:3055] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:42.195348 23721 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:42.196676 23723 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.197252 23723 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.199365 23723 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.167603 23667 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:42.200209 23667 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.201980 23667 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.203076 23690 consensus_queue.cc:260] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.180439 23356 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.169343 23712 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:42.205075 21201 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:42.165092 23718 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:42.206362 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.188184 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:42.207413 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:42.139564 23720 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.220614 21130 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:42.221635 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:42.169200 23754 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:42.225824 21281 raft_consensus.cc:1240] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:42.222877 23296 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:42.227056 23296 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.227386 23296 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:42.230862 23721 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.231885 23721 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:42.216208 21276 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 4 to 7, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:42.235549 21280 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:42.212646 23718 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.238106 23296 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.238854 23718 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:42.223687 23636 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.239526 23636 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.240345 23296 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.241282 23636 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:42.242070 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 98715.8us
I20251214 02:59:42.243513 21247 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:42.212236 23704 log.cc:927] Time spent T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20251214 02:59:42.211489 23723 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.154021 23760 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.256655 23758 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:42.248119 23450 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.261591 23756 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.262619 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.199353 23746 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.270926 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:42.285480 21130 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:42.223649 23550 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:42.207768 23670 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:42.285689 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 69849.2us
W20251214 02:59:42.296080 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:42.297456 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:42.300487 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.300884 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:42.248864 23667 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.301633 21132 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:42.274533 23758 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.312031 23758 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:42.312579 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:42.312727 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.313270 22018 raft_consensus.cc:2368] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
I20251214 02:59:42.313239 21280 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:42.304519 23670 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:42.302454 23402 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.315218 21171 leader_election.cc:400] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
I20251214 02:59:42.314987 23402 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.315502 21171 leader_election.cc:403] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251214 02:59:42.316243 23546 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:42.308115 23450 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.308398 23724 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.308907 23578 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.310814 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.304358 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.323822 21278 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:42.303752 23550 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:42.307874 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:42.327260 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:42.338403 23670 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:42.318593 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 178543us
I20251214 02:59:42.343235 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:42.281499 23723 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.353128 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.354235 21280 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:42.354406 21277 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:42.300782 23292 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20251214 02:59:42.308805 23713 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20251214 02:59:42.323115 23754 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.325999 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.345016 23533 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.319907 21130 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:42.327258 23781 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:42.345213 23732 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:42.349612 23745 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:42.305845 23667 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.350275 23548 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:42.386593 21280 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251214 02:59:42.387135 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 143684us
I20251214 02:59:42.311602 23718 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.396181 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:42.396713 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.397326 21130 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:42.353582 23757 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251214 02:59:42.379483 22015 raft_consensus.cc:1240] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:42.398844 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.381244 23732 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:42.385267 23296 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:42.393636 23497 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.327266 23724 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.395756 23781 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:42.382671 23533 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.453100 23724 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.433697 23296 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.432966 23732 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:42.379206 23754 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.465229 21985 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:42.395704 23548 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.470750 21128 raft_consensus.cc:1217] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Deduplicated request from leader. Original: 5.7->[6.8-6.8] Dedup: 6.8->[]
I20251214 02:59:42.474737 21128 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 02:59:42.428727 23629 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20251214 02:59:42.472103 23548 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.466998 23648 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251214 02:59:42.488534 23783 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20251214 02:59:42.488480 23648 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.489140 23648 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:42.447832 23635 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.491834 23635 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.493693 23635 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.504397 23602 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:42.504932 23602 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:42.507543 23471 raft_consensus.cc:3055] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:42.507987 23471 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:42.508745 23471 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.509742 23471 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:42.511582 23760 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.472460 23670 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:42.501865 23758 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.472887 23785 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:42.468653 23734 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:42.468695 23720 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.472760 23296 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:42.450327 23497 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.468268 23718 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.461884 23550 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.469905 23715 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.468607 23402 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.522614 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 1198) took 1872 ms. Trace:
I20251214 02:59:42.467711 23667 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.466526 23754 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.547155 23781 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.541038 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:42.550091 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 162102us
I20251214 02:59:42.545740 21128 rpcz_store.cc:276] 1214 02:59:40.650028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:40.733697 (+ 83669us) service_pool.cc:225] Handling call
1214 02:59:40.752075 (+ 18378us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:42.470672 (+1718597us) spinlock_profiling.cc:243] Waited 8.99 ms on lock 0x7b5400373320. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef15df0 00007f670ef9fe40 00007f670efa0eb1 00007f670ef9da0f 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c
1214 02:59:42.477106 (+ 6434us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:42.522325 (+ 45219us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400169754. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:42.522485 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1240231808}
I20251214 02:59:42.554154 21278 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:42.520341 23758 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.511731 23648 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.543321 23578 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.584172 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.584602 23533 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.597371 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.598084 21128 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 02:59:42.584792 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.587205 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.602370 21273 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:42.559626 23715 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.603492 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:42.604712 23715 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.605377 23445 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:42.605878 23445 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:42.516683 23712 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:42.606240 23445 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:42.608263 23712 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.524926 23602 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.610622 23602 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.611454 21202 raft_consensus.cc:1240] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 10. Current term is 11. Ops: []
I20251214 02:59:42.596601 21207 raft_consensus.cc:2368] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6.
W20251214 02:59:42.620071 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 184840us
W20251214 02:59:42.624449 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 278658us
I20251214 02:59:42.626106 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:42.641644 21986 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.004s sys 0.013s
W20251214 02:59:42.642017 21986 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.004s sys 0.013s
W20251214 02:59:42.640365 23788 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251214 02:59:42.672067 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:42.673233 21132 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:42.685359 23296 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.672562 23582 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:42.687474 23582 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.686406 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.690076 21206 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:42.691259 23497 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.663506 23546 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5.
W20251214 02:59:42.778499 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 388894us
I20251214 02:59:42.734246 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.779412 21201 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:42.766080 21132 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:42.791800 21247 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:42.779116 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:42.778636 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 140412us
I20251214 02:59:42.787519 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:42.787859 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:42.794024 21985 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:42.712446 23296 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.836732 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:42.844990 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 54320.6us
W20251214 02:59:42.845268 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:42.734930 23648 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:42.734112 23533 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:42.851679 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:42.853780 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 232751us
I20251214 02:59:42.853261 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:42.836081 23652 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:42.844399 23445 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:42.862442 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 67219.7us
I20251214 02:59:42.863507 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.865110 23721 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:42.857044 23497 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251214 02:59:42.856900 23533 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.868671 23550 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:42.869403 23360 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:42.851775 23804 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.873939 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:42.911448 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:42.892314 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 100586us
I20251214 02:59:42.903272 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:42.905720 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:42.909880 21099 leader_election.cc:400] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6.
I20251214 02:59:42.918607 21099 leader_election.cc:403] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:42.919265 21247 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:42.937896 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.938632 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:42.938992 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.939347 21132 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:42.876976 23652 raft_consensus.cc:697] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:42.939966 21274 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:42.940572 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:42.955132 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.955713 21278 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:42.923900 23804 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.956169 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.956769 21128 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:42.957343 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.957520 23804 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.958179 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:42.958609 21204 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:42.958971 23603 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.959108 23802 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:42.959354 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:42.959545 23582 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.962729 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.962849 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:42.963289 21279 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:42.959775 23635 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.959470 23603 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.963812 23635 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.965233 23603 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.965624 23635 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.966946 21278 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:42.968019 23533 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:42.971577 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.972661 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:42.973464 21278 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:42.977001 23776 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:42.977547 23776 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.921281 23543 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:42.979143 23543 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.979910 23776 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:42.981174 23543 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.914777 23550 raft_consensus.cc:697] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:42.982501 23550 consensus_queue.cc:237] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.985581 23804 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:42.986109 23804 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:42.987033 23804 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.987988 23652 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:42.940745 23712 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:42.991634 23712 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:42.993785 23712 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:42.933730 23629 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 6
I20251214 02:59:43.006836 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:43.013504 23807 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:43.014191 23807 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.017699 21125 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:42.975407 21279 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
W20251214 02:59:43.031355 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 76044.6us
W20251214 02:59:43.021656 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.078s sys 0.107s
W20251214 02:59:43.060715 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s user 0.078s sys 0.107s
I20251214 02:59:42.920442 23721 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.088078 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.084625 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.100857 21278 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 02:59:43.105822 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:43.082667 23800 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.137832 21132 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
W20251214 02:59:43.139221 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:43.111507 23732 consensus_meta.cc:287] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.597s user 0.007s sys 0.002s
I20251214 02:59:43.081441 23543 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20251214 02:59:43.145272 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:43.132427 23548 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:43.132725 23560 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.154474 23732 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.152645 23758 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.199054 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:43.200577 21206 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:43.154268 23357 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.202188 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.202550 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:43.205231 23629 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6.
I20251214 02:59:43.207333 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.213734 21275 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:43.207985 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.208153 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.208593 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.209249 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.215577 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.215916 21278 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:43.209440 23550 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:43.215391 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 296289us
I20251214 02:59:43.217469 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:43.219818 21173 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.221359 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.222733 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.226272 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.226836 21281 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:43.229460 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:43.212589 23296 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:43.196144 23560 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.215188 23402 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:43.189729 23548 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.227977 23603 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:43.234392 23560 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.234881 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.239121 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.244784 21199 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:43.250851 21098 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.252404 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.254946 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:43.255532 21277 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 7
I20251214 02:59:43.224792 23715 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251214 02:59:43.239228 23726 log.cc:927] Time spent T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.949s user 0.004s sys 0.010s
I20251214 02:59:43.257925 21986 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:43.224195 23445 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.224370 23546 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:43.214375 23758 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.269420 21098 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:43.270159 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:43.217155 23807 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:43.265432 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.275579 21132 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 9
I20251214 02:59:43.245839 23214 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:43.145164 23800 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.226739 23718 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:43.215055 23357 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.263954 23715 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.225551 23550 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:43.334988 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 56280.6us
I20251214 02:59:43.345846 21986 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:43.269328 23497 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.250275 23648 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20251214 02:59:43.348762 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.269227 23640 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:43.379909 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:43.393414 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:43.363131 21207 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:43.364332 21206 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:43.358503 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.334424 23667 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:43.345162 23764 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:43.344681 23718 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:43.401479 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.418638 21130 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:43.364359 23548 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:43.346988 23715 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:43.406313 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:43.407148 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.410225 21277 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 02:59:43.365022 23445 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:43.404835 23764 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:59:43.425948 23360 consensus_meta.cc:287] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.511s user 0.010s sys 0.000s
I20251214 02:59:43.390908 23543 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.438175 23806 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.438313 23543 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.438858 23806 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.439221 22015 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 601) took 1681 ms. Trace:
I20251214 02:59:43.439617 22015 rpcz_store.cc:276] 1214 02:59:41.757868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:41.768208 (+ 10340us) service_pool.cc:225] Handling call
1214 02:59:41.826942 (+ 58734us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:42.431497 (+604555us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:43.438962 (+1007465us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54005d1954. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:43.439158 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1205843328}
I20251214 02:59:43.440403 23543 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.441107 21208 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:43.441543 23357 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:43.364529 23497 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.398249 21132 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:43.364409 23721 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.405722 23461 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:43.429319 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:43.457819 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.459568 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.484195 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:43.486763 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 57539.4us
I20251214 02:59:43.430284 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:43.365166 23745 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election won for term 5
W20251214 02:59:43.349849 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 79740.8us
I20251214 02:59:43.431145 23764 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:43.435492 23718 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:43.436618 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.516685 21199 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:43.345993 23807 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:43.452530 23550 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:43.454382 23800 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.513914 21206 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:43.560034 21986 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:43.444198 23732 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.539754 23461 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:43.579486 23648 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:43.580408 21130 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:43.579690 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.580202 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.540835 23402 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:43.592259 23648 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:43.613360 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.566915 23357 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.579608 23715 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.567389 23745 raft_consensus.cc:697] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
W20251214 02:59:43.623747 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 395216us
I20251214 02:59:43.647377 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.600980 23714 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.649428 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.651403 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.564994 23807 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.653924 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.658011 21207 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:43.659502 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.617095 23648 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:43.661979 21245 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.663693 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.647008 23550 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:43.623142 23461 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:43.657191 23548 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:43.695780 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.652808 23603 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:43.699368 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.665450 23758 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:43.704766 22017 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.008s sys 0.014s
I20251214 02:59:43.631168 23357 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:43.708357 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 486491us
W20251214 02:59:43.705324 22017 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.595s user 0.008s sys 0.014s
W20251214 02:59:43.705961 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:43.691216 23712 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:43.707985 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.657279 23806 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.712653 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.657567 23714 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.709139 21247 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:43.702502 23776 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:43.704012 23129 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:43.724248 21275 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:43.724690 21275 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:43.707297 23548 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:43.709746 23712 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.725404 21275 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.735317 21130 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:43.736796 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.749629 21275 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:43.739810 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:43.747289 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 370161us
I20251214 02:59:43.749997 21131 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:43.753750 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.754385 21277 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:43.757018 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:43.750847 21133 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:43.758507 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.763806 21130 raft_consensus.cc:2393] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:43.761641 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.765180 21133 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:43.766108 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.767270 23548 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.723320 23776 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.777081 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.777756 21282 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:43.778719 21133 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:43.777436 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.773805 21130 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20251214 02:59:43.774142 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:43.782013 23818 log.cc:927] Time spent T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20251214 02:59:43.782500 23357 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:43.729694 23497 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.750337 23712 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
W20251214 02:59:43.783874 23817 log.cc:927] Time spent T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20251214 02:59:43.784848 23214 consensus_meta.cc:287] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.503s user 0.003s sys 0.004s
I20251214 02:59:43.785284 23828 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:43.785704 23828 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.759491 23543 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:43.786038 23828 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:43.786795 23550 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:43.787072 23816 log.cc:927] Time spent T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20251214 02:59:43.787642 23550 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.788089 23550 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.789884 23550 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:43.792279 21131 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 02:59:43.792906 23603 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.746909 23471 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.794023 23471 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.747021 23640 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.799825 21275 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20251214 02:59:43.807157 21275 pending_rounds.cc:85] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 6
W20251214 02:59:43.783257 23819 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20251214 02:59:43.786347 23543 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.807637 21275 pending_rounds.cc:107] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20251214 02:59:43.807178 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:43.786796 21205 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20251214 02:59:43.846807 21275 raft_consensus.cc:2889] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251214 02:59:43.847124 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:43.848821 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 281769us
I20251214 02:59:43.873967 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
W20251214 02:59:43.808619 21281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.002s sys 0.027s
W20251214 02:59:43.882997 21281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.002s sys 0.027s
I20251214 02:59:43.866905 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.780875 23776 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:43.792795 23296 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:43.885911 21130 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
W20251214 02:59:43.872929 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 211030us
I20251214 02:59:43.854094 23715 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.728382 23745 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.807804 23640 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:43.878417 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 169347us
I20251214 02:59:43.849161 23720 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.850308 23543 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:43.847291 23550 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:43.907728 23720 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.913940 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.914676 21125 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:43.915861 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.917179 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.918221 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.774832 23461 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.874188 23714 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:43.892426 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 96601.2us
I20251214 02:59:43.893186 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:43.849888 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:43.947551 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 54468.4us
I20251214 02:59:43.947067 23764 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.948904 21132 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:43.899189 23807 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:43.950428 23828 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.951375 23714 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:43.951808 23714 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:43.952162 23714 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:43.955008 23546 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:43.957751 21126 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 1857) took 1835 ms. Trace:
I20251214 02:59:43.958173 21126 rpcz_store.cc:276] 1214 02:59:42.122016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.122150 (+ 134us) service_pool.cc:225] Handling call
1214 02:59:42.131406 (+ 9256us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:42.305983 (+174577us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:42.306014 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:42.314222 (+ 8208us) log.cc:844] Serialized 26 byte log entry
1214 02:59:42.429687 (+115465us) raft_consensus.cc:1645] Marking committed up to 2
1214 02:59:42.429728 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:43.957314 (+1527586us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400169254. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:43.957369 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:43.957420 (+ 51us) raft_consensus.cc:1685] finished
1214 02:59:43.957432 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:43.957685 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4756,"raft.queue_time_us":9888,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"spinlock_wait_cycles":389725568,"thread_start_us":115086,"threads_started":1,"wal-append.queue_time_us":180553,"wal-append.run_cpu_time_us":1701,"wal-append.run_wall_time_us":1001728}
I20251214 02:59:43.960372 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.961434 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.966065 21247 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:43.966931 23807 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:43.950497 21173 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.968509 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.859083 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.962420 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.971037 21129 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:43.903002 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.972460 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.973048 21133 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:43.973850 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:43.973187 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 69936.8us
I20251214 02:59:43.903859 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.906716 21130 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:43.974948 21128 raft_consensus.cc:2368] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:43.907752 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.921788 23640 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:43.988436 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.989070 21127 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:43.927162 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.927911 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.989956 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:43.990046 21125 raft_consensus.cc:2368] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6.
I20251214 02:59:43.990883 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.928570 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:43.991844 21275 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:43.992161 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.929298 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.993196 21279 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:43.929932 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:43.993625 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:43.930048 23790 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:43.931360 23839 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:43.931891 23838 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.004190 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.006876 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.008770 21127 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:43.932308 23844 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.009423 23764 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:43.932888 23802 consensus_meta.cc:287] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.973s user 0.004s sys 0.006s
I20251214 02:59:44.010010 23802 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:43.933182 23550 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:44.011476 23550 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.012405 23838 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.933955 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:43.892144 23804 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:43.938093 21201 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:43.927188 23471 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.015658 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.938835 23712 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:43.942754 23648 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:43.947606 21131 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:43.984200 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:43.893556 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:43.989542 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.011643 23543 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.017875 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.018213 21247 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.020573 21173 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.021024 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.021258 21247 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:43.887877 23296 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:44.014750 21274 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:44.021651 21199 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:44.025730 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.027033 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.027731 21199 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:44.000509 23821 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:43.999785 23807 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251214 02:59:44.063995 21207 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251214 02:59:44.066814 21130 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:44.020226 23804 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.025087 23802 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.050271 23826 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:44.050055 23602 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:44.057493 23402 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:44.057945 23764 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:44.098315 23764 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:44.100078 23763 log.cc:927] Time spent T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 1.701s user 0.001s sys 0.005s
I20251214 02:59:44.080008 23804 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:44.108774 23648 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.085155 23548 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:43.404238 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:44.011409 23714 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:44.121910 21208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2640) took 2430 ms (client timeout 1506 ms). Trace:
I20251214 02:59:44.091686 21199 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:44.004720 23738 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20251214 02:59:44.009812 23839 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.125522 21273 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:44.025949 23844 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.121214 21204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2665) took 1249 ms. Trace:
I20251214 02:59:44.131613 21204 rpcz_store.cc:276] 1214 02:59:42.871836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.871933 (+ 97us) service_pool.cc:225] Handling call
1214 02:59:44.121055 (+1249122us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b54001ca054. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:44.121153 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91889408}
I20251214 02:59:44.132877 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:44.064855 23828 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.069538 23823 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:44.122299 21208 rpcz_store.cc:269] 1214 02:59:41.691130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:41.753749 (+ 62619us) service_pool.cc:225] Handling call
1214 02:59:44.121726 (+2367977us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b540008eb54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:44.121821 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3870374784}
W20251214 02:59:44.174059 23841 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20251214 02:59:44.005260 23790 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.190593 23746 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:44.214319 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 193807us
I20251214 02:59:44.215479 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.218397 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:44.202994 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 181794us
I20251214 02:59:44.243242 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:44.219467 21202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2650) took 1646 ms. Trace:
I20251214 02:59:44.243908 21202 rpcz_store.cc:276] 1214 02:59:42.573315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.573453 (+ 138us) service_pool.cc:225] Handling call
1214 02:59:42.573749 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:42.611884 (+ 38135us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:44.219216 (+1607332us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b5400147dd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:44.219395 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86265216}
I20251214 02:59:44.244449 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.124066 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.219892 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:42016 (request call id 879) took 1018 ms. Trace:
I20251214 02:59:44.234988 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.245332 21207 rpcz_store.cc:276] 1214 02:59:43.201082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:44.139379 (+938297us) service_pool.cc:225] Handling call
1214 02:59:44.219832 (+ 80453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:44.237735 23790 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.250764 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:44.239486 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 221776us
I20251214 02:59:44.202109 22014 raft_consensus.cc:1240] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:44.220343 23560 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251214 02:59:44.253661 23550 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:44.253623 23340 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.257146 21247 leader_election.cc:400] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:44.257480 21247 leader_election.cc:403] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251214 02:59:44.258491 21247 leader_election.cc:400] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6.
I20251214 02:59:44.258813 23826 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.259482 23826 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:44.256618 23550 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.261351 23550 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.256212 23635 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:44.263038 23635 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.263521 23635 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.258762 21247 leader_election.cc:403] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251214 02:59:44.256008 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.231783 23828 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:44.272890 21029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 263) took 13483 ms (client timeout 15000 ms). Trace:
I20251214 02:59:44.274986 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:44.239997 23543 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.277526 23839 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.277988 21028 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 3 to 4, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65), VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66), VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67), VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:44.232009 23838 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.292830 23548 consensus_queue.cc:1059] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:44.283092 23635 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.284157 23776 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.231491 23746 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.284929 23828 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.301579 21099 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.296840 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:44.302871 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.304136 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.305145 21099 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:44.285194 23781 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6.
I20251214 02:59:44.262995 22011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44272 (request call id 385) took 1480 ms. Trace:
I20251214 02:59:44.285684 23759 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:44.287860 21281 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 2627) took 1051 ms. Trace:
I20251214 02:59:44.275897 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.291348 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.292212 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.295136 21202 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:44.295303 21247 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.313014 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.317070 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 318) took 3030 ms. Trace:
I20251214 02:59:44.356647 21202 raft_consensus.cc:606] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.265211 23714 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.293526 23690 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:44.358356 21207 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:59:44.368611 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 63505.8us
W20251214 02:59:44.311089 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 257111us
I20251214 02:59:44.297730 21028 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: config changed from index 4 to 7, term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to <none>, 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) changed from NON_VOTER to VOTER. New cstate: current_term: 6 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:44.291179 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 234697us
I20251214 02:59:44.356271 21199 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:44.291072 23712 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.306668 21098 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:44.273310 21029 rpcz_store.cc:269] 1214 02:59:30.785562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:30.785711 (+ 149us) service_pool.cc:225] Handling call
1214 02:59:44.268861 (+13483150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:41.088830 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:41.206839 (+118009us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:41.206899 (+ 60us) write_op.cc:435] Acquired schema lock
1214 02:59:41.206912 (+ 13us) tablet.cc:660] Decoding operations
1214 02:59:41.207512 (+ 600us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:41.207598 (+ 86us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:41.207628 (+ 30us) tablet.cc:683] Acquiring locks for 3 operations
1214 02:59:41.208085 (+ 457us) tablet.cc:699] Row locks acquired
1214 02:59:41.208101 (+ 16us) write_op.cc:260] PREPARE: finished
1214 02:59:41.208256 (+ 155us) write_op.cc:270] Start()
1214 02:59:41.208418 (+ 162us) write_op.cc:276] Timestamp: P: 1765681181208193 usec, L: 0
1214 02:59:41.208435 (+ 17us) op_driver.cc:348] REPLICATION: starting
1214 02:59:41.271399 (+ 62964us) log.cc:844] Serialized 1626 byte log entry
1214 02:59:41.347892 (+ 76493us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f670eb375ab
1214 02:59:41.813490 (+465598us) op_driver.cc:464] REPLICATION: finished
1214 02:59:42.258000 (+444510us) write_op.cc:301] APPLY: starting
1214 02:59:42.258080 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:42.258206 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:42.258225 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:42.258692 (+ 467us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:42.258862 (+ 170us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1214 02:59:42.258906 (+ 44us) write_op.cc:312] APPLY: finished
1214 02:59:42.259100 (+ 194us) log.cc:844] Serialized 35 byte log entry
1214 02:59:42.259255 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:42.259346 (+ 91us) write_op.cc:454] Released schema lock
1214 02:59:42.259388 (+ 42us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:42.259733 (+ 345us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:42.259765 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":537323,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":444272,"apply.run_cpu_time_us":1724,"apply.run_wall_time_us":1725,"num_ops":3,"prepare.queue_time_us":248637,"prepare.run_cpu_time_us":33856,"prepare.run_wall_time_us":350742,"raft.queue_time_us":348681,"raft.run_cpu_time_us":20197,"raft.run_wall_time_us":347096,"replication_time_us":604886,"spinlock_wait_cycles":65511296,"thread_start_us":863993,"threads_started":4,"wal-append.queue_time_us":127386,"wal-append.run_cpu_time_us":60846,"wal-append.run_wall_time_us":1867788}]]}
W20251214 02:59:44.364113 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 88390.1us
I20251214 02:59:44.371197 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:44.334513 21208 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
W20251214 02:59:44.359232 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 64008.2us
I20251214 02:59:44.357012 21205 rpcz_store.cc:276] 1214 02:59:41.286982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:41.287165 (+ 183us) service_pool.cc:225] Handling call
1214 02:59:41.298275 (+ 11110us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:43.847056 (+2548781us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:44.316806 (+469750us) spinlock_profiling.cc:243] Waited 2.49 s on lock 0x7b54001caf54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:44.317005 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5722670464}
I20251214 02:59:44.297061 23723 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.376886 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.378005 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:44.379455 21171 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.379616 21199 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:44.380802 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.305996 23764 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.320214 23828 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.315526 23746 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.356426 21281 rpcz_store.cc:276] 1214 02:59:43.236096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:43.239074 (+ 2978us) service_pool.cc:225] Handling call
1214 02:59:43.245277 (+ 6203us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:43.883367 (+638090us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:43.883397 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:43.889264 (+ 5867us) log.cc:844] Serialized 26 byte log entry
1214 02:59:44.049675 (+160411us) raft_consensus.cc:1645] Marking committed up to 4
1214 02:59:44.049717 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:44.049794 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:44.287651 (+237857us) raft_consensus.cc:1685] finished
1214 02:59:44.287674 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:44.287798 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1460,"raft.queue_time_us":366153,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":2346,"thread_start_us":760194,"threads_started":2,"wal-append.queue_time_us":367262}
I20251214 02:59:44.429131 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.433461 23723 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:44.405349 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:44.430019 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 49290.1us
I20251214 02:59:44.368607 23839 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.357903 21202 raft_consensus.cc:614] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:44.373286 23560 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:44.311281 23402 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:44.382627 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:44.492429 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:44.492791 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 113420us
W20251214 02:59:44.507792 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 125274us
W20251214 02:59:44.492791 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 186228us
I20251214 02:59:44.356525 22011 rpcz_store.cc:276] 1214 02:59:42.783013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.790706 (+ 7693us) service_pool.cc:225] Handling call
1214 02:59:44.230540 (+1439834us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b5800212a98. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2dec 00007f670beb29ab 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f53bd0 00007f6713f534b4 00007f670d4e3834 00007f670d4e37c2
1214 02:59:44.262911 (+ 32371us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7090304}
W20251214 02:59:44.493662 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 96369.3us
I20251214 02:59:44.498771 23856 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.436036 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.313618 23721 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:44.496006 23781 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.515277 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.314292 23635 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:44.499454 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 93195.5us
W20251214 02:59:44.431851 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 62189.2us
I20251214 02:59:44.501757 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.312438 23776 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.378739 23757 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:44.525915 23757 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.504035 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:44.381451 23823 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:44.526304 23757 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.409245 23844 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.547784 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.550151 23844 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:44.552165 21202 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:44.495895 23759 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.312659 23729 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:44.514098 21245 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.505820 23715 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20251214 02:59:44.498672 23640 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:44.302878 23550 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.522505 22017 raft_consensus.cc:1240] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20251214 02:59:44.431902 23533 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20251214 02:59:44.508242 23560 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.524104 23840 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:44.526891 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:44.528247 23814 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.897s user 0.005s sys 0.000s
I20251214 02:59:44.529192 23741 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:44.530898 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.531132 21124 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:44.515347 23721 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.531903 23723 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:44.523231 23781 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.498632 21028 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.574427 23723 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.574718 23357 consensus_queue.cc:1059] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:44.571732 23741 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:44.575811 21203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 2655) took 1937 ms. Trace:
I20251214 02:59:44.533972 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.534463 23823 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.577023 21028 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.536630 22018 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44274 (request call id 543) took 1699 ms. Trace:
I20251214 02:59:44.578276 23768 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.147s
I20251214 02:59:44.566313 23755 consensus_queue.cc:1059] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251214 02:59:44.566521 23828 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 02:59:44.579761 23855 log.cc:927] Time spent T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20251214 02:59:44.576197 21203 rpcz_store.cc:276] 1214 02:59:42.638853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.638980 (+ 127us) service_pool.cc:225] Handling call
1214 02:59:42.649699 (+ 10719us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:42.650054 (+ 355us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:42.650082 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:42.832106 (+182024us) log.cc:844] Serialized 26 byte log entry
1214 02:59:43.257042 (+424936us) raft_consensus.cc:1645] Marking committed up to 3
1214 02:59:43.257078 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:43.257139 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:44.527703 (+1270564us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:44.575525 (+ 47822us) raft_consensus.cc:1685] finished
1214 02:59:44.575547 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:44.575741 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94166,"raft.queue_time_us":11140,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":7855,"spinlock_wait_cycles":6607232,"thread_start_us":383262,"threads_started":1,"wal-append.queue_time_us":661179}
I20251214 02:59:44.580773 21203 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:44.576094 23781 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.537063 23360 raft_consensus.cc:3055] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:44.537696 21205 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:44.582907 23360 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:44.583801 23360 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.585237 23360 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:44.508996 23802 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.512812 23856 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.540814 21207 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:44.544992 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.546001 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.547284 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.381640 23838 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:44.555202 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:44.560990 21199 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:44.515090 21281 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:44.562403 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.614908 21274 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.571007 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:44.615698 21282 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:44.574364 23759 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
W20251214 02:59:44.572727 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 45916.7us
I20251214 02:59:44.561533 23640 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.580853 23550 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.569101 23560 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:44.573988 23721 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:44.537864 21247 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:44.587296 21028 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 7 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.569583 23857 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20251214 02:59:44.594305 23533 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:44.594316 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 61109.7us
I20251214 02:59:44.631302 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:44.587165 22018 rpcz_store.cc:276] 1214 02:59:42.837142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.838360 (+ 1218us) service_pool.cc:225] Handling call
1214 02:59:42.838675 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:42.852105 (+ 13430us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:42.852139 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:43.949739 (+1097600us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b18004af2c0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6714d41f21 00007f6714d4e901 00007f670bebdfe6 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670d89c9b1 00007f670d8abfe2 00007f670efd9851 00007f670ef9e0ba
1214 02:59:43.950088 (+ 349us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b2c0006b218. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670d89c9b1 00007f670d8abfe2 00007f670efd9851 00007f670ef9e0ba 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22
1214 02:59:43.950518 (+ 430us) log.cc:844] Serialized 26 byte log entry
1214 02:59:44.304364 (+353846us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:44.304405 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:44.304466 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:44.536427 (+231961us) raft_consensus.cc:1685] finished
1214 02:59:44.536448 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:44.536563 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4740,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":4532,"spinlock_wait_cycles":11027456,"thread_start_us":288419,"threads_started":1,"wal-append.queue_time_us":539708}
I20251214 02:59:44.578037 23823 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.595988 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:44.599402 23296 consensus_meta.cc:287] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.540s user 0.008s sys 0.004s
I20251214 02:59:44.601854 23450 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:44.644801 21278 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.612466 23781 consensus_queue.cc:1059] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20251214 02:59:44.550668 23844 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.571216 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.617411 21099 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.651456 23844 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.652359 23720 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.653038 23450 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.656872 21206 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:44.631327 23759 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:44.621532 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.613945 23802 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.569440 23840 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.665690 23840 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.591079 23723 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.613464 23838 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.668541 23838 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:44.665620 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:44.632210 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 94428.9us
I20251214 02:59:44.671979 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.636471 23849 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:44.626855 23640 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:44.628460 23648 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:44.614594 23729 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:44.591744 23585 consensus_queue.cc:1059] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20251214 02:59:44.609831 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.683578 21203 raft_consensus.cc:2393] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 5: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:44.626899 23755 consensus_queue.cc:1059] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20251214 02:59:44.659072 21985 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.659516 21126 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:44.657892 21278 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251214 02:59:44.622769 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 67657.1us
I20251214 02:59:44.621325 21245 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:44.661731 21028 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.640097 23550 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:44.628211 23652 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20251214 02:59:44.665031 23349 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251214 02:59:44.665180 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 43738.5us
I20251214 02:59:44.671540 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.647260 23745 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20251214 02:59:44.678108 23781 consensus_queue.cc:1059] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251214 02:59:44.679184 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:44.659409 23732 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 02:59:44.686683 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 65437us
I20251214 02:59:44.687256 21247 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.689589 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.690234 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.719487 21133 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.698876 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.719748 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.720147 21278 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.692307 21028 catalog_manager.cc:5654] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.723343 21028 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 7 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:44.724540 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:44.645586 23296 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.733363 23296 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.735512 23802 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.736084 23721 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.737710 21127 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:44.738016 23721 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.738240 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.735430 23759 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.688370 23823 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:44.690725 23805 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:44.695187 23349 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.744082 23349 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:44.744481 23757 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.746748 21128 raft_consensus.cc:1240] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
W20251214 02:59:44.748435 23804 consensus_meta.cc:287] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.642s user 0.005s sys 0.006s
I20251214 02:59:44.735316 23856 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.748944 23804 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.674826 23849 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:44.750958 23849 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.751479 21201 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:44.751890 21205 raft_consensus.cc:606] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.752216 21205 raft_consensus.cc:614] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:44.715109 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.755482 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.756556 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.757220 21124 raft_consensus.cc:2393] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:44.758241 21124 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:44.705257 23550 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:44.735278 23729 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.759517 23550 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.717245 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.762352 21273 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:44.715653 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.688859 23648 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.729229 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.729775 23764 raft_consensus.cc:3055] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:44.730593 23826 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:44.731076 23715 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.715612 23839 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:44.702813 23445 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20251214 02:59:44.834887 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.835017 23715 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.836021 21203 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:44.837037 23715 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.720919 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.766371 21204 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 8. Current term is 9. Ops: []
I20251214 02:59:44.838939 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:44.839190 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 57659.2us
I20251214 02:59:44.840538 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:44.841634 23860 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:44.781226 23560 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.843297 21202 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:44.823115 23826 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.844870 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.845713 23826 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.845988 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.826612 23720 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251214 02:59:44.848228 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2679) took 1076 ms. Trace:
I20251214 02:59:44.848376 23839 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.848636 21206 rpcz_store.cc:276] 1214 02:59:43.771374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:44.006611 (+235237us) service_pool.cc:225] Handling call
1214 02:59:44.846971 (+840360us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54001bdad4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:44.848172 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1488355072}
I20251214 02:59:44.792937 23838 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.850288 23839 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.777683 21028 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:44.779771 21126 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:44.865039 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.865782 21126 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:44.866859 21028 catalog_manager.cc:5654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:44.868582 21028 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:44.792618 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:44.731447 23840 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.811897 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.736047 23802 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:44.825565 21278 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:44.873531 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.829936 23648 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.850654 23838 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:44.852944 23804 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.855166 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:44.855402 21205 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:44.861995 23560 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.862856 21127 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:44.779505 21208 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:44.771814 21274 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
W20251214 02:59:44.826048 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 48825.1us
I20251214 02:59:44.875046 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:44.893795 23857 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:44.895649 23349 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:44.897684 23349 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:44.897723 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.877665 21273 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:44.898877 23860 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:44.878350 21133 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:44.886162 21276 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 02:59:44.774824 23533 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.108s
I20251214 02:59:44.780342 23764 raft_consensus.cc:740] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:44.899878 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:44.900249 21205 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:44.900578 21133 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:44.900775 21205 raft_consensus.cc:606] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:44.900483 23764 consensus_queue.cc:260] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:44.901499 23764 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.870944 23823 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:44.890071 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:44.895952 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:44.904012 21206 raft_consensus.cc:1240] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20251214 02:59:44.779191 23757 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:44.915216 21127 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 02:59:44.917261 21124 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20251214 02:59:44.888000 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 145335us
I20251214 02:59:44.908728 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:44.913281 21280 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:44.924621 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 237452us
I20251214 02:59:44.925923 21247 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:44.947371 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:45.000980 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 71089.5us
W20251214 02:59:45.001920 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 58644.5us
I20251214 02:59:45.005930 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.006779 21208 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:44.910535 23828 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:44.904493 23402 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:45.009011 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:45.032272 21277 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 02:59:45.028136 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:44.924013 23720 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:45.009665 23340 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20251214 02:59:45.008397 23802 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:45.043263 21205 raft_consensus.cc:1240] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251214 02:59:45.046085 21131 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:45.047255 21098 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:45.054723 21205 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:45.055486 21205 raft_consensus.cc:606] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.056070 21205 raft_consensus.cc:614] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:45.056511 21131 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:45.057109 23402 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:45.058490 23402 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:45.058933 23402 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.003239 23804 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20251214 02:59:45.041647 23720 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:45.063817 23221 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.064152 23720 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:45.065032 21098 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.067267 23757 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.069734 23840 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:45.066318 21098 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:45.073143 23764 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:45.073616 23764 raft_consensus.cc:697] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.073818 23720 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.074900 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.076046 23640 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.077248 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.079160 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:44.927727 23823 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.042182 23296 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.081120 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:45.089959 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.078181 23849 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.014401 23723 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.095757 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.098495 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 02:59:45.091696 22014 raft_consensus.cc:1240] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:45.099267 23112 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:45.131850 23112 raft_consensus.cc:697] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:45.132833 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:45.132647 23112 consensus_queue.cc:237] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.133723 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:45.130503 21208 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
W20251214 02:59:45.092535 23814 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251214 02:59:45.149693 21129 raft_consensus.cc:2393] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:45.158404 21132 raft_consensus.cc:1240] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 8. Ops: []
I20251214 02:59:45.099730 23838 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:45.160801 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.169049 21276 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:45.092193 23840 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20251214 02:59:45.096824 23648 consensus_queue.cc:1059] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20251214 02:59:45.174045 21098 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.178090 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:45.179271 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:45.183401 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:45.184432 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:45.187029 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.190232 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.190743 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.193262 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.194931 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.195845 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.196360 21208 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 4
I20251214 02:59:45.196569 23221 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.197139 23713 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:45.197615 23713 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.172075 23838 raft_consensus.cc:697] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:45.198724 21200 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:45.199270 23838 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.200661 23640 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.204015 23849 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.204546 23497 raft_consensus.cc:3055] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Stepping down as leader of term 10
I20251214 02:59:45.204926 23497 raft_consensus.cc:740] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.205431 23723 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.205638 23497 consensus_queue.cc:260] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.206647 23823 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:45.207298 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:45.209084 21203 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20251214 02:59:45.210433 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.211261 21207 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:45.204102 23402 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.213996 23402 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.193166 23720 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.183776 23764 consensus_queue.cc:237] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.204092 23741 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.218386 21277 raft_consensus.cc:2393] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:45.219277 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.219785 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.222452 23560 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:45.224226 23292 consensus_queue.cc:1059] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:45.225598 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.226531 23560 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251214 02:59:45.230214 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56276.9us
I20251214 02:59:45.231029 23471 raft_consensus.cc:3055] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:45.231415 23471 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.232178 23471 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.233274 23732 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.235898 23840 consensus_queue.cc:1059] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:45.236814 23840 raft_consensus.cc:3055] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:45.237114 23840 raft_consensus.cc:740] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:45.237867 23840 consensus_queue.cc:260] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.239188 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2676) took 1486 ms. Trace:
I20251214 02:59:45.281749 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.282495 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.273100 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:45.266710 21200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2652) took 2691 ms (client timeout 1893 ms). Trace:
I20251214 02:59:45.268188 23550 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:45.286906 23550 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:45.286422 21200 rpcz_store.cc:269] 1214 02:59:42.574897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:42.574998 (+ 101us) service_pool.cc:225] Handling call
1214 02:59:45.266483 (+2691485us) spinlock_profiling.cc:243] Waited 2.6 s on lock 0x7b540024a754. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:45.266622 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5975217920}
I20251214 02:59:45.285631 21280 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:45.287904 23828 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.287797 23550 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.288990 23550 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.253086 23713 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.285399 21273 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.257005 23450 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:45.282114 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.260926 23471 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:45.291853 21282 raft_consensus.cc:2393] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:45.265628 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.254611 23844 raft_consensus.cc:3055] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:45.294916 23844 raft_consensus.cc:740] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:45.295719 23844 consensus_queue.cc:260] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.296638 23844 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:45.285478 21201 rpcz_store.cc:276] 1214 02:59:43.752316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:44.015412 (+263096us) service_pool.cc:225] Handling call
1214 02:59:45.238994 (+1223582us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b540032e6d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 02:59:45.239127 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1690624384}
I20251214 02:59:45.299677 23497 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:45.284938 23840 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
W20251214 02:59:45.298767 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:45.304899 23741 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
W20251214 02:59:45.307220 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 85828us
I20251214 02:59:45.308030 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.277591 21208 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:45.312131 21280 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 02:59:45.321753 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:45.322778 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:45.331080 23723 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.334403 23719 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251214 02:59:45.341478 23881 log.cc:927] Time spent T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.106s user 0.000s sys 0.003s
W20251214 02:59:45.343585 23882 log.cc:927] Time spent T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.115s user 0.002s sys 0.000s
I20251214 02:59:45.349601 23779 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:45.351266 23779 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:45.352144 23779 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.327194 21098 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:45.359143 23755 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:45.353996 21128 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:45.330327 21273 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:45.362432 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:45.356266 23884 log.cc:927] Time spent T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20251214 02:59:45.322968 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:45.358486 23874 log.cc:927] Time spent T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.140s user 0.002s sys 0.000s
I20251214 02:59:45.369022 23719 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:45.359652 23755 raft_consensus.cc:697] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.369375 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.370128 23719 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 02:59:45.370223 23755 consensus_queue.cc:237] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.370777 23723 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:45.371805 21098 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:45.373577 23732 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:45.374040 23732 raft_consensus.cc:697] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.374627 21132 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:45.375114 23732 consensus_queue.cc:237] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.363636 21125 raft_consensus.cc:1240] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20251214 02:59:45.378026 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:45.378257 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:45.365864 23450 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20251214 02:59:45.378284 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:45.383872 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 60976.8us
I20251214 02:59:45.382921 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:45.385090 21248 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.382995 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:45.394013 21129 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:45.396039 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 219022us
I20251214 02:59:45.304912 23880 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20251214 02:59:45.396849 21099 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.399307 21248 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.407042 21248 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:45.407073 21099 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.399425 23840 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.410070 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.399217 23585 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:45.398361 23828 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.410208 23760 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.415992 23779 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.415491 23868 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.408775 23802 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.418381 23800 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:45.464471 23800 raft_consensus.cc:697] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.465440 23800 consensus_queue.cc:237] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.405906 23690 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20251214 02:59:45.468776 21204 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:45.412621 21099 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.432737 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.435034 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.435777 21985 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.474455 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.475199 21279 raft_consensus.cc:2393] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:45.438876 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.475726 21273 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 5: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:45.440717 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.443888 21278 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:45.449115 21276 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 8. Ops: []
I20251214 02:59:45.431895 23823 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.454293 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:45.472231 23112 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.483668 21280 raft_consensus.cc:2368] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:45.426887 23848 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20251214 02:59:45.447475 23867 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:45.492478 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.472348 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.447247 23585 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
W20251214 02:59:45.499110 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 86536.9us
I20251214 02:59:45.503540 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.478955 23445 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:45.505565 23471 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.508008 23550 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:45.510425 21208 raft_consensus.cc:2393] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:45.504698 23743 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 02:59:45.513569 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.520057 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.520871 23112 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:45.520736 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.559089 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:45.559841 21204 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:45.496701 23840 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20251214 02:59:45.518332 23867 raft_consensus.cc:697] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:45.522184 23690 raft_consensus.cc:3060] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:45.562637 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:45.569314 21099 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.572825 21199 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:45.584169 23112 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:45.585335 21173 leader_election.cc:400] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:45.586522 21173 leader_election.cc:403] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20251214 02:59:45.523519 23713 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20251214 02:59:45.523164 23550 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:45.579851 23746 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:45.616405 21276 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:45.589490 23821 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:45.611876 23550 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.655006 21204 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:45.622530 23755 consensus_queue.cc:1059] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20251214 02:59:45.675848 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 82435.6us
I20251214 02:59:45.668876 21276 raft_consensus.cc:1275] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251214 02:59:45.668934 23891 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251214 02:59:45.654702 23221 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:45.674901 23896 log.cc:927] Time spent T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251214 02:59:45.658807 23340 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
W20251214 02:59:45.698874 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 68151.8us
I20251214 02:59:45.706051 21132 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:45.693194 23732 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:45.633685 23867 consensus_queue.cc:237] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.720180 21132 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:45.718241 23804 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251214 02:59:45.716156 23732 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } attrs { replace: true } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.732443 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.733117 21200 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:45.733834 23732 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:45.735229 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:45.735554 23895 log.cc:927] Time spent T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.133s user 0.002s sys 0.000s
I20251214 02:59:45.736063 23844 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:45.704731 23629 consensus_queue.cc:1059] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20251214 02:59:45.755180 23690 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:45.756911 21248 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:45.775880 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:45.699509 23221 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.761758 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:45.763115 23585 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:45.726677 23755 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20251214 02:59:45.779942 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.777073 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.767172 23550 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:45.779172 21278 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:45.778790 21096 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:45.780177 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:45.784482 22014 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:45.789649 21208 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 02:59:45.775071 23823 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:45.790295 23746 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:45.792351 23823 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:45.799494 23823 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:45.800298 23729 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Stepping down as leader of term 8
I20251214 02:59:45.800709 23729 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.801443 23729 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.804723 23221 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:45.813073 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:45.813795 22014 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
W20251214 02:59:45.814734 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 85561.8us
I20251214 02:59:45.815021 21171 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:45.791867 23585 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20251214 02:59:45.811015 23729 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:45.870788 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:45.883450 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 102298us
I20251214 02:59:45.887074 21131 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:45.933930 21200 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:45.895469 21274 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:45.951366 21099 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:45.819033 23560 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:45.842803 23848 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:45.957417 23848 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:45.958473 21133 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:45.958712 23721 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:45.960896 23848 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:45.964910 23800 raft_consensus.cc:3055] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:45.965354 23800 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:45.966148 23800 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:45.967643 23800 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:45.970039 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:45.990283 22018 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
W20251214 02:59:45.970057 23910 log.cc:927] Time spent T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.208s user 0.003s sys 0.000s
I20251214 02:59:45.985666 21200 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:45.863443 23652 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:45.991528 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:46.000897 21199 raft_consensus.cc:1275] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251214 02:59:46.013207 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:45.983094 23560 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:59:45.999409 23903 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20251214 02:59:46.009270 23892 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251214 02:59:45.941099 23550 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20251214 02:59:46.015499 23732 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:46.010952 23720 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20251214 02:59:46.016976 23560 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:46.031675 21199 raft_consensus.cc:1275] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251214 02:59:46.044529 23746 consensus_queue.cc:1048] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:46.012573 23652 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.027499 23732 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:46.028963 23844 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251214 02:59:46.053740 23732 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:46.051512 23292 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20251214 02:59:46.077791 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:46.071331 23652 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.083094 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.084604 21199 raft_consensus.cc:2410] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20251214 02:59:46.086819 21079 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet a3c57218e1ae40dfbdcec8dd228432ca
W20251214 02:59:46.087317 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet a3c57218e1ae40dfbdcec8dd228432ca
I20251214 02:59:46.088307 23560 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.090726 23823 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.091993 21274 raft_consensus.cc:2410] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20251214 02:59:46.096961 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:46.098151 21200 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.094862 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:46.099242 22018 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:46.100755 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:46.102278 23805 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:46.084056 23292 consensus_queue.cc:1048] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20251214 02:59:46.111196 23732 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.112572 21131 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:46.113159 23732 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.116062 21280 raft_consensus.cc:1275] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251214 02:59:46.116088 21274 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20251214 02:59:46.120563 21274 pending_rounds.cc:85] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 4
I20251214 02:59:46.120772 21274 pending_rounds.cc:107] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251214 02:59:46.120988 21274 raft_consensus.cc:2889] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:46.122615 23848 consensus_queue.cc:1048] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:46.124226 23823 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.124593 23221 raft_consensus.cc:993] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:46.125100 23848 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:46.173913 22018 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
W20251214 02:59:46.177997 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 77854.3us
I20251214 02:59:46.194720 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.125023 23221 raft_consensus.cc:1081] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:46.202150 21029 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 7 to 8, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 8 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:46.215274 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:46.216128 22014 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:46.099836 23560 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.266078 21203 raft_consensus.cc:1275] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20251214 02:59:46.259987 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 81236.7us
I20251214 02:59:46.243844 23112 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.279409 23112 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.211321 21029 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
W20251214 02:59:46.271631 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 85437.2us
I20251214 02:59:46.288539 21029 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:46.290588 21029 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 7 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:46.294850 21029 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: config changed from index 4 to 5, term changed from 3 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66), VOTER 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) evicted. New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:46.297451 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.300137 21199 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.298180 21171 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:46.299542 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 281) took 9555 ms. Trace:
I20251214 02:59:46.302649 21029 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 7 to 9, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 9 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:46.304530 21029 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:46.307160 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:46.308461 23112 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.309755 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:46.310451 22018 raft_consensus.cc:2393] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:46.312250 22017 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.313638 21029 catalog_manager.cc:5654] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:46.239405 23823 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.315897 21029 catalog_manager.cc:5654] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:46.323473 21029 catalog_manager.cc:5654] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:46.315558 23560 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.330466 21029 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:46.318945 23719 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:46.323789 23755 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.303337 21028 rpcz_store.cc:276] 1214 02:59:36.744480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:36.806043 (+ 61563us) service_pool.cc:225] Handling call
1214 02:59:46.299485 (+9493442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:45.260871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:45.261786 (+ 915us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:45.261821 (+ 35us) write_op.cc:435] Acquired schema lock
1214 02:59:45.261839 (+ 18us) tablet.cc:660] Decoding operations
1214 02:59:45.263450 (+ 1611us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:45.263536 (+ 86us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:45.263570 (+ 34us) tablet.cc:683] Acquiring locks for 11 operations
1214 02:59:45.264444 (+ 874us) tablet.cc:699] Row locks acquired
1214 02:59:45.264466 (+ 22us) write_op.cc:260] PREPARE: finished
1214 02:59:45.264617 (+ 151us) write_op.cc:270] Start()
1214 02:59:45.264722 (+ 105us) write_op.cc:276] Timestamp: P: 1765681185264567 usec, L: 0
1214 02:59:45.264742 (+ 20us) op_driver.cc:348] REPLICATION: starting
1214 02:59:45.265385 (+ 643us) log.cc:844] Serialized 4490 byte log entry
1214 02:59:45.347187 (+ 81802us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f670eb375ab
1214 02:59:45.720821 (+373634us) op_driver.cc:464] REPLICATION: finished
1214 02:59:45.861541 (+140720us) write_op.cc:301] APPLY: starting
1214 02:59:45.864484 (+ 2943us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:45.907560 (+ 43076us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:45.907583 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:45.908735 (+ 1152us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:45.908953 (+ 218us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
1214 02:59:45.908996 (+ 43us) write_op.cc:312] APPLY: finished
1214 02:59:45.911549 (+ 2553us) log.cc:844] Serialized 83 byte log entry
1214 02:59:45.911753 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:45.924206 (+ 12453us) write_op.cc:454] Released schema lock
1214 02:59:45.943672 (+ 19466us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:45.984620 (+ 40948us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:45.984662 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":90159,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":94655,"apply.run_cpu_time_us":3257,"apply.run_wall_time_us":141662,"num_ops":11,"prepare.queue_time_us":194242,"prepare.run_cpu_time_us":61804,"prepare.run_wall_time_us":257300,"raft.queue_time_us":50924,"raft.run_cpu_time_us":6905,"raft.run_wall_time_us":94770,"replication_time_us":455923,"spinlock_wait_cycles":126775680,"thread_start_us":627928,"threads_started":4,"wal-append.queue_time_us":355211}]]}
I20251214 02:59:46.355943 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:46.360651 21132 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:46.373229 23719 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:46.327606 23823 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.367730 21276 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50516
I20251214 02:59:46.396181 21276 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:46.396600 21276 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:46.339382 23848 consensus_queue.cc:1048] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251214 02:59:46.362469 23755 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.401242 23755 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.403970 23823 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.411331 21200 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:46.374141 23719 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.351181 23560 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.416180 23560 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.422016 21276 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.425747 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:46.425479 21276 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.426519 21277 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:46.423277 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.437873 21208 raft_consensus.cc:2393] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.439743 21248 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:46.439911 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:46.440627 21132 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:46.496845 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:46.428848 23721 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:46.502979 23848 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 02:59:46.494258 23823 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election won for term 3
W20251214 02:59:46.584018 23929 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.282s user 0.002s sys 0.005s
I20251214 02:59:46.520852 23721 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.591188 23721 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.576354 23823 raft_consensus.cc:697] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:46.607198 23823 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.637217 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:46.638392 21247 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:46.639211 23844 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:46.639575 23844 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.639917 23844 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:46.642270 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:46.642977 22017 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:46.646556 23721 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:46.647065 23721 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:46.647465 23721 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:46.653618 23844 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.673087 23844 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:46.654901 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 245066us
I20251214 02:59:46.673982 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:46.666504 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 226860us
I20251214 02:59:46.676656 21096 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:46.679625 23746 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:46.682387 21207 raft_consensus.cc:1275] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251214 02:59:46.683844 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.684415 21201 raft_consensus.cc:3055] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:46.684692 21201 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:46.685375 21201 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.686215 21201 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:46.686335 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:46.686646 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:46.686950 21124 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:46.687268 23848 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.687743 23848 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.689487 23848 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.690243 23828 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.690755 23828 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.692621 23828 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.694557 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.695328 21200 raft_consensus.cc:2393] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.682183 23746 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:46.696125 23746 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:46.701047 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:46.704108 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.704664 21208 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:46.706287 23867 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251214 02:59:46.713227 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:46.715080 23867 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:46.715636 23867 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:46.716027 23867 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:46.682255 23936 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:46.722533 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:46.720842 21124 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:46.724854 21131 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:46.734063 21201 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:46.735610 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:46.744902 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:46.748271 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:46.748903 21126 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
W20251214 02:59:46.757081 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 65849.7us
I20251214 02:59:46.678925 23721 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.722157 23936 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.783710 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:46.797317 21131 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:46.754799 23828 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:46.806144 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.806770 21208 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 02:59:46.809731 23828 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:46.810823 23828 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.798774 23936 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:46.790431 23719 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.815591 23721 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.815790 23719 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.818050 23719 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.831354 21133 raft_consensus.cc:2368] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:46.798314 23823 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.832834 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:46.833398 21278 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:46.837966 23936 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.839581 21278 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 02:59:46.822284 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:46.840624 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.845019 23112 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 02:59:46.845161 21199 raft_consensus.cc:2393] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.887177 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.886083 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:46.918823 22017 raft_consensus.cc:2410] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20251214 02:59:46.854887 21247 leader_election.cc:400] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:46.900883 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:46.919871 21247 leader_election.cc:403] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251214 02:59:46.855978 21281 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:46.922617 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.923163 21199 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
W20251214 02:59:46.921226 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 66522.3us
I20251214 02:59:46.926349 23112 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:46.926784 23112 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Advancing to term 12
I20251214 02:59:46.921564 23844 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 02:59:46.878847 23746 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.938266 23746 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:46.877099 23867 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.941661 22014 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:46.942163 23867 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.941720 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:46.867043 23823 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.944267 23823 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.926738 21098 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:46.947660 23292 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:46.909754 23942 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:46.948429 23942 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.950238 23942 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:46.947520 23797 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:46.953709 21199 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 02:59:46.958804 23292 raft_consensus.cc:697] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:46.959772 23292 consensus_queue.cc:237] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.966913 23906 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:46.967068 22018 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:46.967478 23906 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:46.968305 21096 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:46.969043 23402 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:46.969496 23906 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:46.969511 23402 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:46.970913 23402 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:46.973433 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.974088 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:46.974925 21203 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 9: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:46.978840 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:46.978931 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:46.980494 22012 raft_consensus.cc:2393] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 3: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:46.979768 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:46.984411 21199 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:46.995819 21127 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:46.997805 23797 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:47.002110 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.002876 21208 raft_consensus.cc:2393] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:47.005548 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:47.006129 21208 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:47.002813 21199 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:47.012959 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:47.019255 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:47.020197 21277 raft_consensus.cc:2410] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20251214 02:59:47.020774 21126 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:47.023638 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:47.020359 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:47.053901 23112 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.055199 21200 raft_consensus.cc:2393] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 12: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:47.056712 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.057396 21126 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 11.
I20251214 02:59:47.060945 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.061568 21126 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:47.060242 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:47.062021 23112 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:47.062480 21133 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:46.978785 23936 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:47.073283 21203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:47.074110 21203 raft_consensus.cc:606] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:47.074596 21203 raft_consensus.cc:614] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:47.075130 21133 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
W20251214 02:59:47.078326 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 76383.7us
I20251214 02:59:47.060002 23934 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:47.082976 23934 raft_consensus.cc:697] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:47.083913 23934 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:47.087538 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 74674.6us
I20251214 02:59:47.060741 23840 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:47.089769 21099 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:47.090862 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:47.133348 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:47.140990 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:47.152753 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:47.157226 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 67566.3us
W20251214 02:59:47.094576 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:47.158107 21203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "143de581216646f79bece2b9f1259f64"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:47.197652 21203 raft_consensus.cc:606] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.198024 21203 raft_consensus.cc:614] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251214 02:59:47.144114 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 338054us
I20251214 02:59:47.065699 23760 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251214 02:59:47.205628 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 64740.2us
I20251214 02:59:47.170341 23757 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Advancing to term 12
I20251214 02:59:47.130973 23550 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:47.216619 23867 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:47.228274 23867 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:47.228705 23867 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:47.237686 23790 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:47.238237 23790 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:47.240172 23790 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:47.241119 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:47.202988 23941 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 02:59:47.213368 23760 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.247666 21203 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:47.222891 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:47.257999 23340 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:47.131444 23840 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.262614 23760 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:47.263290 23840 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:47.359445 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 104366us
I20251214 02:59:47.360795 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.361493 21129 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:47.370278 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:47.371024 21203 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:47.365213 23942 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20251214 02:59:47.372385 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:47.253844 23941 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:47.386036 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 111030us
I20251214 02:59:47.392974 23941 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:47.424337 23956 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251214 02:59:47.438513 23947 log.cc:927] Time spent T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251214 02:59:47.441565 23941 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.442699 21278 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Advancing to term 12
I20251214 02:59:47.422312 23760 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.434782 23956 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.427188 23550 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:47.450165 21278 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 12.
I20251214 02:59:47.451543 23956 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:47.454541 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 201137us
I20251214 02:59:47.474821 22014 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20251214 02:59:47.475070 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 180283us
W20251214 02:59:47.488534 23951 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20251214 02:59:47.514065 21173 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:47.490975 23867 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.517984 23867 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:47.518728 21985 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:47.483603 23957 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20251214 02:59:47.530881 21201 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:47.531431 21201 raft_consensus.cc:606] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.531740 21201 raft_consensus.cc:614] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:47.521715 23878 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:47.534799 22014 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:47.540794 23757 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20251214 02:59:47.541587 23952 log.cc:927] Time spent T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.107s user 0.003s sys 0.000s
I20251214 02:59:47.542922 23936 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:47.543421 23936 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.543876 23732 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:47.542873 23760 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:47.534758 23934 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader election won for term 12
I20251214 02:59:47.548558 23934 raft_consensus.cc:697] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:47.530691 23941 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:47.557724 21133 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 12.
I20251214 02:59:47.513339 23550 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.537101 23878 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.562839 23878 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:47.567214 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.571903 21127 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:47.574788 21132 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:47.579391 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:47.579351 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:47.545259 23936 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:47.585965 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:47.587847 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 59790.4us
I20251214 02:59:47.588812 23957 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:47.590416 21280 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 02:59:47.588905 23934 consensus_queue.cc:237] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.600384 21027 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 6 to 7, leader changed from <none> to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.601773 23956 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.602830 21027 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 5 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:47.611536 21275 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:47.610508 21129 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
W20251214 02:59:47.621472 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 83465.6us
I20251214 02:59:47.621999 21027 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.627921 21027 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 6 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:47.610165 23878 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:47.639688 23963 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:47.642316 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:47.643122 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:47.684216 21278 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:47.643898 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.644943 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 301) took 3060 ms. Trace:
I20251214 02:59:47.637166 21027 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 3 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.687047 21133 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:47.686693 21027 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 4 to 5, term changed from 3 to 5, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.688326 21027 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:47.686124 21029 rpcz_store.cc:276] 1214 02:59:44.585060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:44.585221 (+ 161us) service_pool.cc:225] Handling call
1214 02:59:47.644863 (+3059642us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:46.744932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:46.811122 (+ 66190us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:46.811159 (+ 37us) write_op.cc:435] Acquired schema lock
1214 02:59:46.811177 (+ 18us) tablet.cc:660] Decoding operations
1214 02:59:46.812138 (+ 961us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:46.812217 (+ 79us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:46.812244 (+ 27us) tablet.cc:683] Acquiring locks for 11 operations
1214 02:59:46.813099 (+ 855us) tablet.cc:699] Row locks acquired
1214 02:59:46.813117 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:46.813255 (+ 138us) write_op.cc:270] Start()
1214 02:59:46.813356 (+ 101us) write_op.cc:276] Timestamp: P: 1765681186813208 usec, L: 0
1214 02:59:46.813376 (+ 20us) op_driver.cc:348] REPLICATION: starting
1214 02:59:46.814010 (+ 634us) log.cc:844] Serialized 4398 byte log entry
1214 02:59:47.242296 (+428286us) op_driver.cc:464] REPLICATION: finished
1214 02:59:47.429140 (+186844us) write_op.cc:301] APPLY: starting
1214 02:59:47.429228 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:47.429451 (+ 223us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:47.429558 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:47.430593 (+ 1035us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:47.430845 (+ 252us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
1214 02:59:47.434000 (+ 3155us) write_op.cc:312] APPLY: finished
1214 02:59:47.464438 (+ 30438us) log.cc:844] Serialized 83 byte log entry
1214 02:59:47.477765 (+ 13327us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:47.487342 (+ 9577us) write_op.cc:454] Released schema lock
1214 02:59:47.500685 (+ 13343us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:47.515683 (+ 14998us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:47.519559 (+ 3876us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4669,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":186553,"apply.run_cpu_time_us":3145,"apply.run_wall_time_us":81920,"num_ops":11,"prepare.queue_time_us":240173,"prepare.run_cpu_time_us":3934,"prepare.run_wall_time_us":132783,"replication_time_us":428776,"thread_start_us":411394,"threads_started":2}]]}
I20251214 02:59:47.690019 21027 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.695209 21027 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.697338 21027 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 2, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 2 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:47.700513 21027 catalog_manager.cc:5654] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:47.701822 21203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:47.702373 21203 raft_consensus.cc:606] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.702384 21027 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 3 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 5 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:47.704520 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:47.705235 21131 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:47.709066 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:47.696609 23550 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:47.700901 23956 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:47.731079 23960 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:47.731590 23960 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.731937 23960 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:47.755043 21114 tablet_service.cc:1552] Processing DeleteTablet for tablet d55cde2542d544c5b99e6113b95910b8 with delete_type TABLET_DATA_TOMBSTONED (TS 721d604130924d939d291fe3eef8f6d1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47052
I20251214 02:59:47.728550 23550 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:47.810271 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.788218 22018 raft_consensus.cc:1240] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:47.811039 23340 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:47.811972 23550 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.812691 23340 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.814741 23340 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:47.819473 23972 tablet_replica.cc:333] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: stopping tablet replica
I20251214 02:59:47.812009 21202 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:47.809332 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 64648.4us
I20251214 02:59:47.830629 21199 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:47.831207 21127 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:47.836795 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:47.837487 21278 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:47.839874 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.841589 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:47.845639 21200 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:47.839699 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:47.855481 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:47.856163 21127 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:47.857362 21201 raft_consensus.cc:1275] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 02:59:47.833899 23732 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:47.859686 23732 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.861721 23732 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:47.852741 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.865942 21131 raft_consensus.cc:2410] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20251214 02:59:47.866326 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:47.869616 21202 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:47.859609 23972 raft_consensus.cc:2243] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:47.873757 21204 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:47.874316 21204 raft_consensus.cc:606] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.875653 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:47.878058 21275 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:47.879369 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:47.879578 21127 raft_consensus.cc:2468] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 02:59:47.881191 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.881950 21202 raft_consensus.cc:2393] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:47.884860 21985 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:47.885569 23292 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:47.865398 23915 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:47.892167 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:47.843129 23340 consensus_queue.cc:1059] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20251214 02:59:47.894191 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:47.895071 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:47.899664 23972 raft_consensus.cc:2272] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:47.892127 23915 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.900568 23915 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:47.903008 23974 log.cc:927] Time spent T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20251214 02:59:47.911171 23957 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:47.922647 23648 raft_consensus.cc:2804] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:47.923171 23648 raft_consensus.cc:697] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:47.924096 23648 consensus_queue.cc:237] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:47.928755 23970 log.cc:927] Time spent T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251214 02:59:47.892804 23292 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:47.935989 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.937817 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:47.934870 23292 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:47.938835 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:47.893343 23878 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:47.914546 23960 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:47.942111 23941 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251214 02:59:47.933634 23936 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20251214 02:59:47.962102 21199 raft_consensus.cc:2368] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10.
I20251214 02:59:47.946066 23954 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:47.943701 23585 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.018s
I20251214 02:59:47.945595 23497 consensus_queue.cc:1048] T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
W20251214 02:59:47.977751 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 92971.9us
W20251214 02:59:47.980356 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47734.4us
I20251214 02:59:47.981537 21099 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:47.988408 21201 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:48.015075 23729 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:47.996471 23942 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
W20251214 02:59:48.153263 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 91549.4us
I20251214 02:59:48.286602 23221 raft_consensus.cc:3055] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:48.287163 23221 raft_consensus.cc:740] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:48.288089 23221 consensus_queue.cc:260] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:48.289227 23221 raft_consensus.cc:3060] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:48.121939 23729 raft_consensus.cc:697] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:48.310025 23960 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:48.341526 23729 consensus_queue.cc:237] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:48.375030 23975 log.cc:927] Time spent T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.389s user 0.001s sys 0.000s
W20251214 02:59:48.408500 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 376346us
W20251214 02:59:48.414706 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:48.407752 23292 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:48.427886 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 214239us
I20251214 02:59:48.433840 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:48.434470 21129 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:48.446352 21275 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20251214 02:59:48.457301 21129 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 10.
I20251214 02:59:48.471781 23292 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:48.487968 23961 raft_consensus.cc:993] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
W20251214 02:59:48.649847 23915 consensus_meta.cc:287] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.749s user 0.010s sys 0.000s
I20251214 02:59:48.669687 23915 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:48.671046 21207 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:48.749142 23915 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:48.760872 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 557041us
I20251214 02:59:48.846971 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:48.836299 23550 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:48.855861 21202 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:48.857570 21248 leader_election.cc:400] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10.
I20251214 02:59:48.867791 23961 raft_consensus.cc:1081] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:48.868594 23972 ts_tablet_manager.cc:1916] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:48.872677 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:48.875504 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:48.876163 21282 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:48.880791 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:48.881491 21205 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 6: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:48.873412 21127 raft_consensus.cc:2393] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:48.881914 21128 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:48.887603 21131 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:48.888140 21131 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:48.888481 21131 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:48.891913 21129 raft_consensus.cc:1275] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20251214 02:59:48.893363 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:48.897891 21128 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:48.903116 21173 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:48.904632 21281 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20251214 02:59:48.904980 23112 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:48.905483 23112 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:48.906019 23980 raft_consensus.cc:993] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20: : Instructing follower 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:48.906451 23980 raft_consensus.cc:1081] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Signalling peer 721d604130924d939d291fe3eef8f6d1 to start an election
I20251214 02:59:48.906260 23112 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:48.862687 23550 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:48.909327 23550 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:48.912803 21099 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:48.904989 21281 pending_rounds.cc:85] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 3
I20251214 02:59:48.921283 21281 pending_rounds.cc:107] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20251214 02:59:48.922509 21281 raft_consensus.cc:2889] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:48.921124 21131 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:48.935467 23954 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:48.947204 21128 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:48.948477 23957 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:48.890061 23941 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:48.950066 23941 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:48.951925 23941 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:48.954909 23729 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:48.967271 21131 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:48.968411 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 48097.4us
I20251214 02:59:48.968873 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:48.979688 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49455.6us
I20251214 02:59:48.975086 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:48.982959 21205 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
W20251214 02:59:49.012730 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 53264.5us
I20251214 02:59:48.984577 21205 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:49.024423 24002 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20251214 02:59:49.025918 21203 raft_consensus.cc:1275] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251214 02:59:48.979872 23982 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20251214 02:59:49.029111 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:49.029397 23982 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:49.029772 23982 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:49.034000 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 53378.7us
I20251214 02:59:49.041323 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:49.043993 23585 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:49.046105 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:49.046674 21274 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:49.015887 21205 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.051978 21203 raft_consensus.cc:1275] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251214 02:59:49.052385 21203 pending_rounds.cc:85] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 5
I20251214 02:59:49.052640 21203 pending_rounds.cc:107] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251214 02:59:49.052148 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:49.052873 21203 raft_consensus.cc:2889] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:49.046097 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:49.069233 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251214 02:59:49.071177 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 89132.2us
I20251214 02:59:49.076133 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:49.090627 21274 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:49.092022 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:49.092882 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 171673us
I20251214 02:59:49.070852 23550 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:49.096882 23745 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20251214 02:59:49.097043 23729 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:49.098237 23954 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 02:59:49.216635 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 84922.3us
I20251214 02:59:49.132812 23993 consensus_queue.cc:1048] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20251214 02:59:49.235191 23221 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:48.948954 23957 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.186426 23729 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:49.030521 23982 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.249226 23112 consensus_queue.cc:1048] T 01028fc01cbf4f4c88f7e36696d08d03 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20251214 02:59:49.249884 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 182330us
I20251214 02:59:49.287184 21133 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:47062
I20251214 02:59:49.287943 21133 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:49.288283 21133 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:49.288865 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 250188us
W20251214 02:59:49.313275 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 243206us
W20251214 02:59:49.314460 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:49.319773 21133 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.338249 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:49.338833 21204 raft_consensus.cc:3055] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:49.339102 21204 raft_consensus.cc:740] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:49.340338 21133 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:49.347608 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:49.339747 21204 consensus_queue.cc:260] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.348243 21281 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:49.348667 21204 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:49.381989 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:49.398602 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 02:59:49.439311 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:49.444293 21204 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
W20251214 02:59:49.444733 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 98616.4us
W20251214 02:59:49.445576 21282 consensus_meta.cc:287] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.562s user 0.012s sys 0.000s
I20251214 02:59:49.462070 21282 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:49.453729 21281 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
W20251214 02:59:49.464148 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 119969us
I20251214 02:59:49.463601 23996 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.470641 23996 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.427232 23963 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 02:59:49.446250 24007 log.cc:927] Time spent T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20251214 02:59:49.465981 23550 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.469132 23942 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.355010 23729 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.469216 23878 consensus_queue.cc:1048] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20251214 02:59:49.487524 23963 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:49.537006 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 91326.7us
W20251214 02:59:49.541172 24012 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.183s user 0.000s sys 0.000s
I20251214 02:59:49.521209 23972 ts_tablet_manager.cc:1929] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20251214 02:59:49.543954 23972 log.cc:1199] T d55cde2542d544c5b99e6113b95910b8 P 721d604130924d939d291fe3eef8f6d1: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-0-root/wals/d55cde2542d544c5b99e6113b95910b8
I20251214 02:59:49.511778 23550 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.514915 23942 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.535719 23954 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.551326 23954 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.552430 23942 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:49.553615 23954 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:49.554260 23957 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:49.555150 23957 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:49.555620 23957 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.548159 23963 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:49.548820 23996 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:49.557035 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:49.557382 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:49.568768 24010 log.cc:927] Time spent T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
W20251214 02:59:49.563063 24021 log.cc:927] Time spent T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20251214 02:59:49.563015 24015 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.561823 23550 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:49.571069 24018 log.cc:927] Time spent T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20251214 02:59:49.574326 23972 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47052 (request call id 979) took 1872 ms. Trace:
I20251214 02:59:49.573793 24015 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.575075 23972 rpcz_store.cc:276] 1214 02:59:47.724228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:47.754734 (+ 30506us) service_pool.cc:225] Handling call
1214 02:59:47.819088 (+ 64354us) ts_tablet_manager.cc:1151] Deleting tablet d55cde2542d544c5b99e6113b95910b8
1214 02:59:47.819178 (+ 90us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 02:59:49.326100 (+1506922us) tablet_metadata.cc:641] Metadata flushed
1214 02:59:49.502984 (+176884us) tablet_metadata.cc:641] Metadata flushed
1214 02:59:49.574208 (+ 71224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34671,"tablet-delete.queue_time_us":63270,"thread_start_us":67314,"threads_started":1}
I20251214 02:59:49.574553 23957 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:49.577065 24015 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:49.613258 21098 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:49.631063 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 112043us
I20251214 02:59:49.652160 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:49.653580 21282 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:49.656728 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:49.658412 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 02:59:49.659332 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 46578.3us
I20251214 02:59:49.659704 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:49.669307 21205 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:49.683516 23732 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:49.683804 23848 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
W20251214 02:59:49.283548 23941 consensus_queue.cc:1175] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "668ad4f30bd649c88a034bbab9f13b20" responder_term: 10 status { last_received { term: 8 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251214 02:59:49.684583 23848 raft_consensus.cc:697] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:49.687741 23732 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.686225 23848 consensus_queue.cc:237] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.708649 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:49.711539 21173 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:49.712853 24038 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:49.713900 24038 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:49.715498 24038 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:49.705139 23957 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.723954 23957 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:49.723812 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:49.738298 21017 catalog_manager.cc:4985] TS 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): tablet d55cde2542d544c5b99e6113b95910b8 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:49.740190 23957 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:49.740638 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:49.980748 23550 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:49.895093 24015 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 02:59:49.884577 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:50.036332 21205 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
W20251214 02:59:50.038522 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:49.722747 23732 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:50.071574 23956 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:50.072783 23956 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:50.268960 24015 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:50.271661 24015 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:50.690219 23996 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:50.691192 23996 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:50.693514 23996 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:50.731356 23996 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:50.903100 23996 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } attrs { replace: true } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:50.945978 21202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 403) took 1185 ms. Trace:
I20251214 02:59:50.946576 21202 rpcz_store.cc:276] 1214 02:59:49.760666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:49.881634 (+120968us) service_pool.cc:225] Handling call
1214 02:59:50.945279 (+1063645us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:50.945614 (+ 335us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:50.945631 (+ 17us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:50.945683 (+ 52us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:50.945733 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:50.945773 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:50.945900 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:50.040439 23550 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.037412 23550 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.038004 23550 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.038733 23550 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.040935 23550 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.050113 23550 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.050918 23550 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.052608 23550 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.052983 23550 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.053459 23550 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.055114 23550 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.055562 23550 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.056063 23550 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.057462 23550 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.057816 23550 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.058246 23550 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.059602 23550 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.067545 24048 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.068272 24048 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.070276 24048 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.071417 24048 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.071664 23957 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.072041 24048 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.072485 23957 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.074084 24048 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.074631 23957 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:51.074882 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:51.075315 23957 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.075960 23957 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.077893 23957 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:51.075320 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:51.108657 23957 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.109319 23957 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.111624 23957 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.114130 21278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 412) took 1067 ms. Trace:
I20251214 02:59:51.114586 21278 rpcz_store.cc:276] 1214 02:59:50.047007 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:50.049750 (+ 2743us) service_pool.cc:225] Handling call
1214 02:59:50.050078 (+ 328us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:51.112189 (+1062111us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:51.112220 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:51.112271 (+ 51us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:51.112301 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:51.113748 (+ 1447us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001d4a54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 02:59:51.113799 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:51.113942 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2442253696}
I20251214 02:59:51.161819 23957 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.162689 23957 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.165014 23957 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.228788 23957 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.229564 23957 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.229914 24048 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.230660 24048 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.231932 23957 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.232501 23957 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.232627 24048 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.233395 23550 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.233099 23957 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.233877 23550 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:51.235245 23957 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.235433 23550 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.233183 24048 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.237139 24048 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.238433 21273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 411) took 1202 ms. Trace:
I20251214 02:59:51.238837 21273 rpcz_store.cc:276] 1214 02:59:50.035923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:50.037889 (+ 1966us) service_pool.cc:225] Handling call
1214 02:59:50.042696 (+ 4807us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 02:59:50.050915 (+ 8219us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:50.050943 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 02:59:50.052414 (+ 1471us) raft_consensus.cc:1645] Marking committed up to 6
1214 02:59:50.052453 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:50.052508 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:51.238275 (+1185767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 02:59:51.238871 24048 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.243394 24048 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.243988 24048 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.245929 24048 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.251863 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 402) took 1583 ms. Trace:
W20251214 02:59:51.252241 21098 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.606s user 0.005s sys 0.017s
W20251214 02:59:51.252665 21098 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.606s user 0.005s sys 0.018s
I20251214 02:59:51.255298 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.255987 21129 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:51.252270 21201 rpcz_store.cc:276] 1214 02:59:49.667953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:49.668139 (+ 186us) service_pool.cc:225] Handling call
1214 02:59:49.668734 (+ 595us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:49.669384 (+ 650us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:49.669409 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:49.670139 (+ 730us) log.cc:844] Serialized 26 byte log entry
1214 02:59:49.705432 (+ 35293us) raft_consensus.cc:1645] Marking committed up to 5
1214 02:59:49.705498 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:49.705578 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:51.251488 (+1545910us) raft_consensus.cc:1685] finished
1214 02:59:51.251519 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:51.251688 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7727,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":285,"thread_start_us":34786,"threads_started":1,"wal-append.queue_time_us":356327,"wal-append.run_cpu_time_us":2127,"wal-append.run_wall_time_us":1049797}
I20251214 02:59:51.259709 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.260403 21129 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:51.263588 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.265731 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.266721 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.267501 21124 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:51.268446 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.269344 21129 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:51.270520 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.272795 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.274482 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.276820 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.277482 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.277840 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.277683 21125 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:51.278648 21128 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:51.280318 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:51.307575 21096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.649s user 0.000s sys 0.037s
I20251214 02:59:51.309201 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.309919 21128 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
W20251214 02:59:51.315496 21096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.658s user 0.001s sys 0.037s
I20251214 02:59:51.315740 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.325275 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:51.326306 23957 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:51.327872 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 1.67041e+06us
I20251214 02:59:51.329566 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.330555 21133 raft_consensus.cc:2410] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20251214 02:59:51.344497 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.346273 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.347968 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:51.348317 23221 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.004s sys 0.016s
I20251214 02:59:51.348655 21133 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:51.350836 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.351912 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.356019 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.356649 21273 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
W20251214 02:59:51.348727 23221 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.305s user 0.021s sys 0.038s
I20251214 02:59:51.361501 21247 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:51.366717 23941 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:51.367175 23941 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.367554 23941 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:51.369509 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.370873 23957 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:51.371274 23957 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.371666 23957 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:51.376263 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.716s user 0.002s sys 0.014s
W20251214 02:59:51.379024 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.719s user 0.004s sys 0.015s
W20251214 02:59:51.379415 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 1.71914e+06us
I20251214 02:59:51.389809 21274 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:51.397472 23963 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.398197 23963 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.401062 23963 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:51.411864 21247 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.713s (SENT) [suppressed 17 similar messages]
I20251214 02:59:51.417001 23941 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.424716 21124 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20251214 02:59:51.435689 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.436167 23957 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.439568 23292 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.440152 23292 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:51.441222 21205 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 2803) took 2470 ms (client timeout 2182 ms). Trace:
W20251214 02:59:51.441672 21205 rpcz_store.cc:269] 1214 02:59:48.970954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:48.974828 (+ 3874us) service_pool.cc:225] Handling call
1214 02:59:49.098447 (+123619us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b18001ec660. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efd9496 00007f670ef135be 00007f670ef94c1e 00007f670ef9645e 00007f670efa1efc 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c
1214 02:59:49.667913 (+569466us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b580095f0d0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670eefa0da 00007f670ef7c177 00007f670ef94c67 00007f670ef9645e 00007f670efa1efc 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c
1214 02:59:51.441115 (+1773202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":396062848}
I20251214 02:59:51.442301 23292 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.447177 21247 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.448199 24049 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.448812 24049 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.449270 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.451006 23550 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:51.451154 24049 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.451402 23550 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.451793 23550 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:51.453891 24049 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.454465 24049 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.456959 24049 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.457837 24049 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.438463 23957 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:51.472652 24011 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.964s user 0.009s sys 0.034s
W20251214 02:59:51.473052 24011 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.998s user 0.009s sys 0.035s
I20251214 02:59:51.462435 24048 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:51.458423 24049 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.485157 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.487630 23993 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.488324 23993 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.492663 23941 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.500609 23729 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.500895 23956 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.501463 23956 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.501353 23729 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.503810 24049 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.504328 24049 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.504784 24049 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.505093 23996 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.505633 23996 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:51.505952 23996 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.501974 23956 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.525203 23729 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.506284 23996 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:51.548794 23941 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.549364 23941 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.545393 23729 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:51.525231 23956 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.550707 23729 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.551358 23941 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.546118 24049 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.559437 24060 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.560166 24060 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.562227 24060 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.559333 23729 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.564174 24058 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.543869 23550 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.564734 24058 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.567147 23956 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.567667 23956 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.568542 24058 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.566670 23993 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.570770 23996 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.570222 23550 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.578729 23956 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 02:59:51.582835 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:51.590077 23956 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.597600 23996 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:51.609010 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56070.3us
W20251214 02:59:51.655653 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:51.603140 23550 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:51.672551 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 178881us
I20251214 02:59:51.601257 23738 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.604612 23747 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.680023 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.606145 23978 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.619517 23745 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.653060 24066 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:51.686877 24068 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:51.687520 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:51.619684 23956 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.689849 23747 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.691828 24065 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.692601 23402 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:51.692519 24065 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.693011 23402 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.693374 23402 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:51.694067 23747 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.695370 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.696002 21132 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 02:59:51.697757 23956 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.694516 24065 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.689368 24068 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.686300 23550 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.698988 24068 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:51.699054 24069 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.698987 24066 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.699021 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:51.698957 24003 log.cc:927] Time spent T 2ca27c31805c4e55898fe4f60468ffa6 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 2.572s user 0.006s sys 0.010s
W20251214 02:59:51.703766 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 43479.7us
I20251214 02:59:51.688369 23978 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.704046 23745 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.708302 23745 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.711683 24064 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.712293 24064 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.714243 24064 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.705837 24069 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.716677 24069 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.717883 24064 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.686048 23738 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.718604 24064 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.720172 23738 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.721038 24064 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.736876 23550 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.740232 23402 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.743350 24066 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.743894 23978 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:51.744593 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 57155.1us
I20251214 02:59:51.745339 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.746729 23963 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:51.747123 23963 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.747543 23963 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:51.748675 23747 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.754400 23990 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.755110 23990 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.757520 23990 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.758070 23990 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.758889 23745 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.758816 23997 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.772318 23997 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.755823 24048 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.761144 23982 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:51.742316 23402 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.764444 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.765539 24034 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.709457 23956 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:51.767266 23936 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.769615 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.774704 24033 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:51.774873 23960 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.766166 23963 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.781191 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.784080 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.774814 23934 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.754383 23747 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.776403 24048 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:51.785888 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.814899 23960 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.816169 23797 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:51.816787 23797 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.804702 24034 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.819284 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.820259 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.820562 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.784530 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.807091 21204 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:51.763710 24068 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.777369 23956 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.808370 24033 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.836362 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.839541 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:51.839877 23999 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.840255 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:51.847720 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 105927us
I20251214 02:59:51.841084 23221 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.841346 21279 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:51.841872 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.842453 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.848663 21276 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:51.843220 23963 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.844038 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:51.851747 21281 raft_consensus.cc:2393] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:51.821255 21201 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:51.821303 21275 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:51.780030 23745 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.758826 23990 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.847314 21206 raft_consensus.cc:2376] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7. Re-sending same reply.
I20251214 02:59:51.854149 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.862018 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.851701 21282 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:51.846733 23956 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:51.786844 23936 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.868820 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:51.820602 24034 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.861287 23221 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.817931 23960 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.869434 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:51.875620 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.868938 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.869467 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.905982 21129 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:51.906875 21126 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:51.865082 23997 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.868471 22020 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:51.877498 24073 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:51.907779 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.908172 24073 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.908759 23936 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.906272 21127 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:51.851994 23934 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.909935 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:51.910336 24073 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.878600 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.910873 23934 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.878847 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.879002 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.879411 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.881402 23745 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.849550 24038 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.909015 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:51.913125 21125 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:51.886826 23797 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.888392 24033 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.872864 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.916837 24038 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.918351 23221 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.918840 24038 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.889747 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.890786 21205 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:51.851059 23999 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.922485 23999 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.900122 21273 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20251214 02:59:51.923005 21273 pending_rounds.cc:85] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Aborting all ops after (but not including) 5
I20251214 02:59:51.923252 21273 pending_rounds.cc:107] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20251214 02:59:51.923496 21273 raft_consensus.cc:2889] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:51.921096 23747 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.924903 23957 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:51.925335 23957 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.928522 23957 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:51.910957 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.931748 23990 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:51.882086 24048 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:51.874343 21247 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:51.902848 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.904593 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:51.877691 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:51.908991 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.907788 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.878131 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.916188 21202 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:51.936983 21202 raft_consensus.cc:606] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:51.937300 21099 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:51.937315 21202 raft_consensus.cc:614] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:51.939265 21282 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
W20251214 02:59:51.916435 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800331560 after lost signal to thread 19984
W20251214 02:59:51.944247 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800091340 after lost signal to thread 21065
W20251214 02:59:51.945406 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08003146c0 after lost signal to thread 21157
W20251214 02:59:51.945676 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800327fc0 after lost signal to thread 21158
W20251214 02:59:51.946728 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080033d9e0 after lost signal to thread 21232
W20251214 02:59:51.946978 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800395ea0 after lost signal to thread 21233
I20251214 02:59:51.949136 21125 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
W20251214 02:59:51.949384 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080038a980 after lost signal to thread 21306
W20251214 02:59:51.949869 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2460 after lost signal to thread 21307
I20251214 02:59:51.951939 21129 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:51.961191 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.961608 21127 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
W20251214 02:59:51.964912 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08003c3ba0 after lost signal to thread 22044
W20251214 02:59:51.965333 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08003ce0c0 after lost signal to thread 22045
W20251214 02:59:51.978097 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080012a860 after lost signal to thread 23857
I20251214 02:59:51.978192 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 02:59:51.977109 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 109786us
I20251214 02:59:51.979308 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.979588 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.981770 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.978703 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.980479 21131 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
W20251214 02:59:51.980854 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08001d8660 after lost signal to thread 23878
I20251214 02:59:51.983636 21099 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:51.984275 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800037880 after lost signal to thread 23942
W20251214 02:59:51.984586 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08003ba760 after lost signal to thread 23954
W20251214 02:59:51.984840 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800223ca0 after lost signal to thread 23964
I20251214 02:59:51.948124 23934 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 02:59:51.985088 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08002f0040 after lost signal to thread 23972
W20251214 02:59:51.987027 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800243880 after lost signal to thread 23975
W20251214 02:59:51.987349 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080037b0a0 after lost signal to thread 24002
I20251214 02:59:51.985741 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.988023 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.988626 21282 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 02:59:51.988519 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.985603 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.967840 23402 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:51.993249 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:51.992312 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080037aa00 after lost signal to thread 24010
I20251214 02:59:51.985493 23957 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:51.997776 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.996950 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:51.997133 23797 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:51.998950 23906 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251214 02:59:51.999126 23797 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.000128 24070 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:52.000537 24070 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:52.000878 24070 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:51.993861 21277 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 1.
I20251214 02:59:51.993541 23402 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.001580 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.001861 23402 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:51.993290 24034 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.003028 24034 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:51.994623 23934 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.000780 23956 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:52.004998 24034 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.996865 23999 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:51.976073 24066 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.006117 21281 raft_consensus.cc:1240] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:52.006415 24066 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.005786 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 157) took 6274 ms. Trace:
I20251214 02:59:52.003238 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.008417 24066 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.006191 23999 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.005766 23956 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.010221 23999 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.001271 23797 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.011724 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.012331 21207 raft_consensus.cc:1813] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Handling vote request from an unknown peer 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.012778 21207 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:52.014295 23934 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.014834 24048 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.018863 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.019469 21202 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5751534b61ed4568abd62057238d5fda"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:52.020025 21202 raft_consensus.cc:606] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.020352 21202 raft_consensus.cc:614] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:52.021260 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.022874 21202 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:52.023411 21202 raft_consensus.cc:606] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.023736 21202 raft_consensus.cc:614] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:52.024989 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.025558 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.026139 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.026748 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.026911 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.027698 21199 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:52.027848 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.028084 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.028458 21200 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:52.028941 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:52.029358 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.030534 21273 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:52.030718 23963 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:52.033084 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.033234 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.034283 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.035482 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.035602 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.035775 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.036665 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.036864 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.037979 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.034241 21208 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:51.992646 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:51.996217 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:51.996474 24068 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:51.997334 23221 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.041111 23221 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.041363 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae31f8b580840eb863e4d400367c1be with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50512
I20251214 02:59:52.043000 23221 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.043179 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.029475 21199 raft_consensus.cc:606] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.034431 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.044672 21199 raft_consensus.cc:614] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251214 02:59:52.045117 21173 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:52.030045 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:51.992396 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.044693 21247 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 02:59:51.994972 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08003a36c0 after lost signal to thread 24013
I20251214 02:59:52.048504 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.049309 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.049484 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:52.049513 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 62534.5us
I20251214 02:59:52.046386 21245 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.052335 21245 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.058601 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.053934 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.059238 21130 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 2.
I20251214 02:59:52.059087 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.059827 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.054589 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.055236 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.055581 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.055728 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.056299 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.062085 23221 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:52.062345 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.062814 23221 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.063174 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.064591 23221 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.065393 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.066135 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.066241 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.056517 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.056321 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.049885 24068 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:52.057348 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.057622 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.057726 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.058169 21199 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:52.058760 21124 raft_consensus.cc:1240] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:52.060668 23797 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:52.061645 23934 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:52.065423 21096 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:52.079813 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.068137 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.052397 23956 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 02:59:52.070734 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080037ae60 after lost signal to thread 24014
W20251214 02:59:52.075596 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 83265.9us
I20251214 02:59:52.082134 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.082522 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:52.013288 21028 catalog_manager.cc:5654] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.081401 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.103466 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.087594 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.107105 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:52.107753 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800328ac0 after lost signal to thread 24018
W20251214 02:59:52.108078 21062 debug-util.cc:398] Leaking SignalData structure 0x7b08002e7c40 after lost signal to thread 24020
I20251214 02:59:52.108175 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.105141 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:52.108942 21098 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.106508 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.112756 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 02:59:52.113933 21062 debug-util.cc:398] Leaking SignalData structure 0x7b0800360100 after lost signal to thread 24021
W20251214 02:59:52.114231 21062 debug-util.cc:398] Leaking SignalData structure 0x7b080031f0a0 after lost signal to thread 24022
I20251214 02:59:52.112011 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.112622 23934 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.117658 23934 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.118120 23934 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:52.118793 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.083050 23797 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.120347 21028 catalog_manager.cc:5654] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.124506 23402 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.074155 23957 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.126852 21199 raft_consensus.cc:606] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.131289 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.127537 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.131665 21199 raft_consensus.cc:614] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:52.127110 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.103293 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.127200 23797 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.136312 23755 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:52.136725 23755 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.137077 23755 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:52.138789 24038 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:52.141449 23936 consensus_queue.cc:1059] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:52.142328 23402 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.156834 21096 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:52.158537 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 49685us
I20251214 02:59:52.077227 24068 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.159250 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:52.122316 21028 catalog_manager.cc:5654] T 0ba6dea370cb421894536a9154735510 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 2, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 2 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.160728 23906 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:52.162508 23906 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251214 02:59:52.163101 23906 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.170905 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.126641 24037 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:52.176985 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.143971 24076 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
W20251214 02:59:52.184738 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 49917.1us
I20251214 02:59:52.184774 21028 catalog_manager.cc:5654] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 7 to 8, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 8 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.189841 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.190409 21199 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:52.144345 24062 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:52.127926 24066 consensus_queue.cc:1059] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20251214 02:59:52.193567 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.194082 21985 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:52.197336 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 115624us
I20251214 02:59:52.142469 23936 consensus_queue.cc:1059] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20251214 02:59:52.198927 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.127285 23934 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.202736 21201 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:52.175681 24037 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.176587 24068 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:52.203747 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.184532 23995 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:52.205591 21985 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:52.184439 24076 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.194260 21028 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } }
I20251214 02:59:52.194702 24062 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.209208 23745 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.216243 23960 raft_consensus.cc:3055] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:52.216727 23960 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.217257 23907 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.217646 23960 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.206611 24037 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:52.208154 23995 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:52.007941 21027 rpcz_store.cc:276] 1214 02:59:45.731198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:45.731747 (+ 549us) service_pool.cc:225] Handling call
1214 02:59:45.856116 (+124369us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b54000cadd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670efa6bba 00007f6714d2513c 00007f6714e1f44e 00007f671331d4a4 00007f671331d432 00007f671331d361 00007f671331d2dc 00007f671331c592 00007f670d2dd55c 00007f670d2dc996
1214 02:59:52.005567 (+6149451us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:48.084404 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:48.382979 (+298575us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:48.383018 (+ 39us) write_op.cc:435] Acquired schema lock
1214 02:59:48.383035 (+ 17us) tablet.cc:660] Decoding operations
1214 02:59:48.384056 (+ 1021us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:48.384118 (+ 62us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:48.384147 (+ 29us) tablet.cc:683] Acquiring locks for 12 operations
1214 02:59:48.750461 (+366314us) tablet.cc:699] Row locks acquired
1214 02:59:48.750479 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:48.750669 (+ 190us) write_op.cc:270] Start()
1214 02:59:48.750786 (+ 117us) write_op.cc:276] Timestamp: P: 1765681188750616 usec, L: 0
1214 02:59:48.750806 (+ 20us) op_driver.cc:348] REPLICATION: starting
1214 02:59:48.751519 (+ 713us) log.cc:844] Serialized 4813 byte log entry
1214 02:59:49.463790 (+712271us) op_driver.cc:464] REPLICATION: finished
1214 02:59:51.366873 (+1903083us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670bed2c1c 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beef7f2 00007f67129d6e40
1214 02:59:51.478696 (+111823us) write_op.cc:301] APPLY: starting
1214 02:59:51.478816 (+ 120us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:51.479138 (+ 322us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:51.479171 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:51.483206 (+ 4035us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:51.510853 (+ 27647us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
1214 02:59:51.510904 (+ 51us) write_op.cc:312] APPLY: finished
1214 02:59:51.511350 (+ 446us) log.cc:844] Serialized 89 byte log entry
1214 02:59:51.599679 (+ 88329us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:51.603541 (+ 3862us) write_op.cc:454] Released schema lock
1214 02:59:51.606487 (+ 2946us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:51.672742 (+ 66255us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:51.674387 (+ 1645us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6863,"reported_tablets":22,"spinlock_wait_cycles":186428672,"child_traces":[["op",{"apply.queue_time_us":1988134,"apply.run_cpu_time_us":39324,"apply.run_wall_time_us":193496,"num_ops":12,"prepare.queue_time_us":134141,"prepare.run_cpu_time_us":11386,"prepare.run_wall_time_us":796651,"raft.queue_time_us":318818,"raft.run_cpu_time_us":45210,"raft.run_wall_time_us":2124277,"replication_time_us":693276,"spinlock_wait_cycles":21443456,"thread_start_us":2480024,"threads_started":5,"wal-append.queue_time_us":177535,"wal-append.run_cpu_time_us":31006,"wal-append.run_wall_time_us":1413036}]]}
I20251214 02:59:52.228152 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.213346 21028 catalog_manager.cc:5654] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.230520 21171 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.213881 24062 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:52.235646 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.227916 23995 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:52.228340 23745 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:52.242132 21247 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.233561 23797 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.242115 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.235405 23960 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:52.244050 21247 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.250145 21171 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.274767 21280 raft_consensus.cc:1275] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:52.274797 21985 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.241276 23402 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.262053 24082 tablet_replica.cc:333] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:59:52.287194 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.287789 24082 raft_consensus.cc:2243] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:52.288311 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.288515 24082 raft_consensus.cc:2272] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:52.289340 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.290900 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.291924 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.292871 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.292925 21199 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 02:59:52.273434 23957 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.293854 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.294879 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.264009 24076 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.295877 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.296375 24076 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:52.298774 24083 log.cc:927] Time spent T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20251214 02:59:52.298868 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.299538 21099 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:52.299691 24085 consensus_queue.cc:1175] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e36caefa37d47a5968cffd9e270044f" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20251214 02:59:52.301270 21208 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:52.301594 24071 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:52.302073 24071 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:52.279485 23997 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.302894 24071 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.262315 23755 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.280211 24073 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.274109 23999 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.306257 23755 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.281805 24070 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.283766 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.286178 24075 consensus_queue.cc:1059] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20251214 02:59:52.286607 23934 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.264772 23982 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:52.306923 21129 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:52.241914 23745 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.320178 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.290347 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.244014 21028 catalog_manager.cc:5654] T 205cf8366aa44b27a349916da03cff46 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 8, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 8 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.320178 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.276634 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.298691 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:52.324865 21132 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:52.325649 21273 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:52.315711 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.321364 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.322098 23982 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:52.334770 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 58196.1us
I20251214 02:59:52.335592 21247 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.324753 23745 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:52.339659 24082 ts_tablet_manager.cc:1916] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:52.344085 24068 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.345202 23978 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:52.347438 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:52.347615 23978 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:52.348385 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:52.348618 23978 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:52.348999 23978 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.349795 23978 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.351233 23738 consensus_queue.cc:1059] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:52.352243 23747 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:52.352581 23747 raft_consensus.cc:740] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.353528 24070 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.353327 23747 consensus_queue.cc:260] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.354597 23995 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.346288 24068 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.356410 23995 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.360524 23946 raft_consensus.cc:3055] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:52.368736 23946 raft_consensus.cc:740] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.369571 23946 consensus_queue.cc:260] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.370683 23957 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:52.370985 23946 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:52.384217 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.385051 21132 raft_consensus.cc:2393] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:52.386377 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.363206 23747 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:52.364737 23978 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:52.365900 21274 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:52.333310 23982 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:52.392437 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:52.407392 24037 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.411121 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.395459 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:52.425997 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 133651us
I20251214 02:59:52.333401 21028 catalog_manager.cc:5654] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.435945 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.435825 21028 catalog_manager.cc:5654] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.436676 23797 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:52.437052 23797 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.437095 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.437548 23797 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:52.414923 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.438369 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.438877 21129 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 02:59:52.401625 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:52.439139 23221 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.439555 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.430377 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:52.434892 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.439755 23221 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.440644 24088 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:52.440752 21173 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.440280 21275 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:52.441588 23999 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:52.441941 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.442674 23221 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:52.441159 24088 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:52.431938 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.432832 21248 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.441195 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.434907 23934 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20251214 02:59:52.451339 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.455540 23956 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.457345 24037 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.460386 23402 consensus_queue.cc:1048] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:52.461757 24062 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.462692 21986 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:52.463537 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43934.7us
I20251214 02:59:52.465390 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.466122 24073 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.446599 24088 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:52.388535 23957 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.467278 23957 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:52.470084 24075 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.470800 24075 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:52.472510 23738 raft_consensus.cc:3055] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:52.472842 23738 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.473632 23738 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.424144 24048 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.437774 21028 catalog_manager.cc:5654] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.480145 21028 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.481704 21028 catalog_manager.cc:5654] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 2 to 3, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 3 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.463783 24062 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.488163 21203 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:52.490126 21202 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:52.496112 21275 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 02:59:52.507447 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.390690 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.504469 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.478287 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:52.506738 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.507828 21986 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.512991 21986 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 02:59:52.514261 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 72372.3us
I20251214 02:59:52.517939 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:52.519246 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 128619us
I20251214 02:59:52.520803 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.521394 21098 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:52.523258 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:52.524923 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.531186 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.533340 21028 catalog_manager.cc:5654] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:52.534225 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.533564 23221 consensus_queue.cc:1059] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:52.536695 21126 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:52.538754 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.540802 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.512068 23402 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.543638 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.543761 23221 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:52.544339 23221 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.520560 23747 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.545873 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.545476 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.523793 24068 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.548175 21099 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.549001 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.547928 23221 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:52.551081 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.529296 23963 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.532850 24058 raft_consensus.cc:2804] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:52.554685 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.547077 21028 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.556172 21130 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:52.541714 23960 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.558378 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:52.564684 23221 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:52.556232 23755 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.544023 24088 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.569307 24058 raft_consensus.cc:697] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:52.569865 24088 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:52.570264 24058 consensus_queue.cc:237] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.546749 24048 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.556865 23997 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.586557 21028 catalog_manager.cc:5654] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 4, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.587816 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.594641 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.600251 21028 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 8, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 8 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:52.604043 21173 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.604897 23221 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.606156 24076 consensus_queue.cc:1059] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:52.608075 24076 raft_consensus.cc:3055] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Stepping down as leader of term 2
I20251214 02:59:52.608475 24076 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.609462 24076 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.610682 24076 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:52.615617 23797 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.617602 23797 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.545543 24082 ts_tablet_manager.cc:1929] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
W20251214 02:59:52.623499 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 44127.8us
W20251214 02:59:52.627060 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 194307us
I20251214 02:59:52.627938 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.639014 24048 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.640816 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.642825 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:52.643281 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 89912.3us
W20251214 02:59:52.642886 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 116619us
I20251214 02:59:52.646749 21247 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.621380 24085 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.654075 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.655164 21127 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:52.656141 21247 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.656246 24095 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
W20251214 02:59:52.656978 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 143021us
I20251214 02:59:52.653160 24082 log.cc:1199] T 6ae31f8b580840eb863e4d400367c1be P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/6ae31f8b580840eb863e4d400367c1be
I20251214 02:59:52.658726 24095 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:52.659698 24095 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.660977 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 6ae31f8b580840eb863e4d400367c1be (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:52.663870 23745 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.665884 23745 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.642186 23738 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:52.667116 22012 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 2. Current term is 3. Ops: []
I20251214 02:59:52.670758 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.671283 24068 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.641713 23982 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.673326 23982 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.641848 23999 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.618532 23957 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.648702 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.650295 24048 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.651551 23956 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.657891 21986 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.679602 21248 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.694154 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.701730 21986 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.705394 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.707338 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.708314 21986 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.680999 23982 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 02:59:52.717717 23982 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.718169 23982 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:52.710708 23978 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:52.711196 24073 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.712360 24088 consensus_queue.cc:1059] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:52.719966 24088 consensus_queue.cc:1243] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 721d604130924d939d291fe3eef8f6d1 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251214 02:59:52.720325 24048 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.715381 23957 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:52.726876 21986 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:52.704470 24068 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 02:59:52.701503 23738 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:52.730126 21248 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 02:59:52.734336 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 78275.4us
I20251214 02:59:52.736043 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:52.738121 21247 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.739321 21247 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.731447 24073 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.733419 23978 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.741812 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.742767 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.745554 21131 raft_consensus.cc:2393] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 3: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:52.734129 24068 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:52.748314 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.769058 24068 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:52.747713 23957 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:52.735436 24098 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.775135 24104 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.775596 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.776414 24104 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.775167 24048 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.777658 24048 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:52.778016 21248 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.744494 24085 raft_consensus.cc:3055] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:52.778870 24085 raft_consensus.cc:740] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.780045 21248 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.779810 24085 consensus_queue.cc:260] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.781119 21248 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.776973 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:52.783429 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.782202 23956 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:52.794018 21204 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 02:59:52.795668 21986 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.781136 24085 raft_consensus.cc:3060] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:52.800916 21096 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.795307 24095 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.803112 23985 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251214 02:59:52.803308 24095 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 02:59:52.803690 24095 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:52.803759 23946 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:52.797719 24098 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 02:59:52.804302 23946 raft_consensus.cc:697] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.804570 24098 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20251214 02:59:52.815496 21131 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:52.804060 24095 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:52.827771 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.828913 22015 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 02:59:52.827153 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.831677 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:52.832449 23982 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.833928 21207 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:52.835254 24071 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:52.835453 23745 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20251214 02:59:52.836419 23745 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:52.836745 23745 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:52.837011 23946 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.837538 23745 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:52.854031 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 70661.8us
I20251214 02:59:52.854053 23957 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:52.855780 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.857913 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.860558 21248 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 02:59:52.863831 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:52.826853 23936 consensus_queue.cc:1059] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20251214 02:59:52.835768 23956 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:52.865711 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.870018 23936 raft_consensus.cc:3055] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:52.870429 23936 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:52.871457 21207 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 02:59:52.871249 23936 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.872424 23936 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:52.864457 24098 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:52.874662 23956 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.875594 23982 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.876580 23956 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:52.864185 24111 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:52.879112 21248 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.881191 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:52.883201 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:52.885224 23956 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 02:59:52.864826 24110 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:52.890269 24037 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:52.881538 24111 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:52.897110 24111 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:52.897466 24111 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:52.909763 21248 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.855760 23957 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.922477 21248 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.926487 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:52.881707 24068 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.918012 24095 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:52.998240 24095 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:52.954349 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:52.970914 23745 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:52.935647 24114 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:52.971192 21274 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:52.975610 23811 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 02:59:52.981115 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 60353.8us
W20251214 02:59:52.981892 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 55476.1us
I20251214 02:59:53.008488 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:52.982373 23982 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 02:59:52.984926 24111 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:52.929944 24098 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:53.004912 21204 raft_consensus.cc:1240] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 1. Current term is 2. Ops: []
I20251214 02:59:53.013840 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.017287 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.019194 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.009198 23811 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:53.029067 23811 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:53.016965 23956 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251214 02:59:53.025259 23982 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:53.038447 24110 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 02:59:53.020074 24071 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20251214 02:59:53.016820 24068 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:53.043802 23907 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:53.044323 23907 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:53.045152 23907 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.047175 23585 consensus_queue.cc:1059] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251214 02:59:53.048484 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:53.048797 23585 raft_consensus.cc:3055] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 02:59:53.049335 23585 raft_consensus.cc:740] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:53.051067 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:53.052670 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.050395 23585 consensus_queue.cc:260] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.062525 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:53.063266 21202 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 02:59:53.065237 23907 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:53.013659 24095 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251214 02:59:53.074887 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.062441 23585 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 1 FOLLOWER]: Advancing to term 2
I20251214 02:59:53.076481 21281 raft_consensus.cc:2393] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 10: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:53.079213 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.081266 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.082410 21274 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.084322 21099 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:53.089511 21202 raft_consensus.cc:1275] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.093535 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.094893 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.095570 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.096465 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:53.096019 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.099786 21274 raft_consensus.cc:1240] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:53.103503 21274 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:53.104499 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.105947 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.104760 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.107136 21278 raft_consensus.cc:1240] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 02:59:53.110986 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.112264 21273 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:53.113396 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:53.114952 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:53.117587 21204 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:53.118687 21273 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9. Ops: []
I20251214 02:59:53.120146 21277 raft_consensus.cc:1240] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20251214 02:59:53.095748 24099 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:53.097391 24110 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:53.127876 21279 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:53.091914 24073 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20251214 02:59:53.118685 24111 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:53.130393 24110 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.137202 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:53.142114 21131 raft_consensus.cc:1240] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 02:59:53.124140 24037 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20251214 02:59:53.149767 24075 raft_consensus.cc:3055] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:53.125708 24099 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:53.172622 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.175477 21279 raft_consensus.cc:1240] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 02:59:53.170312 21027 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.177934 21173 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.182799 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.183586 23221 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 02:59:53.186157 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.185547 23936 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 02:59:53.181722 24095 consensus_queue.cc:1059] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20251214 02:59:53.189677 24095 raft_consensus.cc:3055] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:53.190049 24095 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:53.165861 24075 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:53.190868 24095 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.191154 24075 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.192030 24095 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:53.192240 24110 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:53.192569 24075 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:53.177223 24099 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:53.207037 24073 consensus_queue.cc:1059] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:53.208092 24073 raft_consensus.cc:3055] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:53.208424 24073 raft_consensus.cc:740] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:53.208806 22015 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 02:59:53.209192 24073 consensus_queue.cc:260] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.216063 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:53.182005 23963 consensus_queue.cc:1048] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20251214 02:59:53.210704 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:53.217538 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:53.213532 21128 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:53.213930 24099 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:53.186249 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.215415 24110 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251214 02:59:53.217420 24111 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 02:59:53.219273 24068 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:53.218479 21027 catalog_manager.cc:5654] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 7 to 9, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 9 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.223539 24088 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 02:59:53.218721 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.230780 21248 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.219345 23963 consensus_queue.cc:1243] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer 668ad4f30bd649c88a034bbab9f13b20 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251214 02:59:53.233471 21245 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:53.232383 21131 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.235667 21247 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:53.235487 21027 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 6 to 7, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.235598 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:53.238116 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.240558 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.241418 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.241950 21248 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:53.231721 24068 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 02:59:53.243036 21203 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.237352 24111 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:53.247613 24088 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:53.237495 23956 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:53.253175 21203 pending_rounds.cc:85] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 3
I20251214 02:59:53.249038 24111 raft_consensus.cc:697] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:53.247756 24098 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 02:59:53.248978 23999 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:53.242887 23982 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:53.254947 21203 pending_rounds.cc:107] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251214 02:59:53.258033 21203 raft_consensus.cc:2889] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:53.258103 23982 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:53.259039 23982 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.243927 24076 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:53.263149 24111 consensus_queue.cc:237] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.255596 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:53.266441 23956 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:53.266652 23999 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251214 02:59:53.267416 23956 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.243803 23957 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251214 02:59:53.270622 24114 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:53.274911 24073 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:53.255699 21027 catalog_manager.cc:5654] T 9fdf7633f868447a8abd2adfe9b04fc5 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 4, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 4 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.255584 23221 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:53.277990 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:53.291473 24076 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251214 02:59:53.294770 23999 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Stepping down as leader of term 8
I20251214 02:59:53.295210 23999 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:53.295929 23999 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.297215 23999 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:53.305680 23221 consensus_queue.cc:1059] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:53.319480 24088 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:53.320585 24088 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:53.320921 24088 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:53.310465 24076 raft_consensus.cc:3055] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 LEADER]: Stepping down as leader of term 10
I20251214 02:59:53.321584 24076 raft_consensus.cc:740] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:53.321681 24088 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.322615 21128 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:53.322399 24076 consensus_queue.cc:260] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.323741 24076 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:53.324029 24066 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20251214 02:59:53.336654 21128 consensus_queue.cc:237] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.341174 24099 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:53.342136 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 264) took 6194 ms. Trace:
I20251214 02:59:53.342499 21028 rpcz_store.cc:276] 1214 02:59:47.148237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:47.148418 (+ 181us) service_pool.cc:225] Handling call
1214 02:59:53.342076 (+6193658us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:52.695419 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:52.696095 (+ 676us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:52.696152 (+ 57us) write_op.cc:435] Acquired schema lock
1214 02:59:52.696171 (+ 19us) tablet.cc:660] Decoding operations
1214 02:59:52.697531 (+ 1360us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:52.697644 (+ 113us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:52.697673 (+ 29us) tablet.cc:683] Acquiring locks for 16 operations
1214 02:59:52.699043 (+ 1370us) tablet.cc:699] Row locks acquired
1214 02:59:52.699074 (+ 31us) write_op.cc:260] PREPARE: finished
1214 02:59:52.699266 (+ 192us) write_op.cc:270] Start()
1214 02:59:52.699442 (+ 176us) write_op.cc:276] Timestamp: P: 1765681192699186 usec, L: 0
1214 02:59:52.699471 (+ 29us) op_driver.cc:348] REPLICATION: starting
1214 02:59:52.818696 (+119225us) log.cc:844] Serialized 6358 byte log entry
1214 02:59:53.031324 (+212628us) op_driver.cc:464] REPLICATION: finished
1214 02:59:53.151283 (+119959us) write_op.cc:301] APPLY: starting
1214 02:59:53.151371 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:53.151682 (+ 311us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:53.151794 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:53.157317 (+ 5523us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:53.157617 (+ 300us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
1214 02:59:53.157661 (+ 44us) write_op.cc:312] APPLY: finished
1214 02:59:53.157974 (+ 313us) log.cc:844] Serialized 113 byte log entry
1214 02:59:53.158134 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:53.158292 (+ 158us) write_op.cc:454] Released schema lock
1214 02:59:53.158341 (+ 49us) write_op.cc:341] FINISH: Updating metrics
1214 02:59:53.158743 (+ 402us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:53.158772 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28683,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":119708,"apply.run_cpu_time_us":3613,"apply.run_wall_time_us":7434,"num_ops":16,"prepare.queue_time_us":75836,"prepare.run_cpu_time_us":22962,"prepare.run_wall_time_us":188572,"raft.queue_time_us":142884,"raft.run_cpu_time_us":26600,"raft.run_wall_time_us":85641,"replication_time_us":330755,"thread_start_us":329186,"threads_started":4,"wal-append.queue_time_us":64585}]]}
I20251214 02:59:53.362807 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:53.363591 21279 raft_consensus.cc:2393] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 10: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:53.364372 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:53.368918 21124 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.348340 23936 consensus_queue.cc:1059] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20251214 02:59:53.361291 24088 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:53.396525 21207 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.397820 23956 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:53.411039 23221 raft_consensus.cc:3055] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 02:59:53.411516 23221 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:53.412276 23221 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:53.413599 23221 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:53.416038 21205 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251214 02:59:53.421074 24075 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:53.433378 21200 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 02:59:53.442028 21099 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:53.452019 24070 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:53.459199 24070 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:53.460173 24070 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:53.488972 21027 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 2 to 3, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 3 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.500216 21131 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:53.501613 23982 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20251214 02:59:53.518700 24123 log.cc:927] Time spent T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20251214 02:59:53.523350 21027 catalog_manager.cc:5654] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 6, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.548723 22015 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:53.558916 21274 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 02:59:53.556035 24100 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:53.561815 22015 raft_consensus.cc:1275] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 02:59:53.577874 21204 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251214 02:59:53.582324 24100 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:53.592305 21030 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 8 to 9, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 9 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.597558 21030 catalog_manager.cc:5654] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.570322 24070 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20251214 02:59:53.619261 24087 consensus_queue.cc:1048] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:53.623643 21030 catalog_manager.cc:5654] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 9, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 9 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:53.634079 21030 catalog_manager.cc:5654] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.653620 24087 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.656148 21131 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:53.657626 23956 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:53.660877 21204 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.665529 21207 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:53.671562 22015 raft_consensus.cc:2955] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 4 to 6, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.673808 21207 raft_consensus.cc:1275] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:53.675352 23957 consensus_queue.cc:1048] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:53.692929 24100 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Committing config change with OpId 7.9: config changed from index 7 to 9, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.696080 21275 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 7 to 9, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.697151 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3fd36aadbf4841c9a26e48e96430c009 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:53.702400 21131 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:53.703694 24111 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:53.722524 21027 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 4 to 6, VOTER 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) evicted. New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.729573 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 302) took 5767 ms. Trace:
I20251214 02:59:53.712486 22012 raft_consensus.cc:2955] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 7 to 9, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.734333 21017 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a3c57218e1ae40dfbdcec8dd228432ca with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251214 02:59:53.743379 21264 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd36aadbf4841c9a26e48e96430c009 with delete_type TABLET_DATA_TOMBSTONED (TS 6e36caefa37d47a5968cffd9e270044f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50512
I20251214 02:59:53.750648 21204 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20251214 02:59:53.729943 21030 rpcz_store.cc:276] 1214 02:59:47.962537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:48.006959 (+ 44422us) service_pool.cc:225] Handling call
1214 02:59:53.674731 (+5667772us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:53.674759 (+ 28us) write_op.cc:454] Released schema lock
1214 02:59:53.729512 (+ 54753us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:53.660102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:53.660603 (+ 501us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:53.660634 (+ 31us) write_op.cc:435] Acquired schema lock
1214 02:59:53.660646 (+ 12us) tablet.cc:660] Decoding operations
1214 02:59:53.661197 (+ 551us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:53.661250 (+ 53us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:53.661274 (+ 24us) tablet.cc:683] Acquiring locks for 4 operations
1214 02:59:53.661646 (+ 372us) tablet.cc:699] Row locks acquired
1214 02:59:53.661664 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:53.661782 (+ 118us) write_op.cc:270] Start()
1214 02:59:53.661875 (+ 93us) write_op.cc:276] Timestamp: P: 1765681193661741 usec, L: 0
1214 02:59:53.661894 (+ 19us) op_driver.cc:348] REPLICATION: starting
1214 02:59:53.662432 (+ 538us) log.cc:844] Serialized 1706 byte log entry
1214 02:59:53.665295 (+ 2863us) op_driver.cc:464] REPLICATION: finished
1214 02:59:53.665722 (+ 427us) write_op.cc:301] APPLY: starting
1214 02:59:53.665795 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:53.665924 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:53.666008 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:53.666472 (+ 464us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:53.666646 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1214 02:59:53.666696 (+ 50us) write_op.cc:312] APPLY: finished
1214 02:59:53.666890 (+ 194us) log.cc:844] Serialized 41 byte log entry
1214 02:59:53.667027 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:53.667171 (+ 144us) write_op.cc:454] Released schema lock
1214 02:59:53.667226 (+ 55us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1631,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":1792,"apply.run_wall_time_us":1792,"num_ops":4,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":2601,"prepare.run_wall_time_us":4112,"replication_time_us":2424}]]}
I20251214 02:59:53.752914 24068 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20251214 02:59:53.774112 21207 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:53.775646 24111 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:53.798717 23907 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:53.799235 23907 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.807845 23907 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:53.809265 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:53.809922 22015 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:53.811352 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.811949 21279 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:53.812570 21096 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:53.813369 23906 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:53.813720 23906 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:53.814050 23906 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:53.823920 21028 catalog_manager.cc:5654] T b98a871e4a86480da87fed8511a68575 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.825733 21028 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 9 to 10, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 10 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:53.830551 21028 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:53.834483 21028 catalog_manager.cc:5654] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:53.851547 22015 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20251214 02:59:53.847321 24073 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:53.853960 24073 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:53.854713 24142 tablet_replica.cc:333] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: stopping tablet replica
I20251214 02:59:53.855942 24073 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:53.860491 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:53.861155 21124 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 8.
I20251214 02:59:53.862362 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:53.863691 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:53.864295 21279 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 8.
I20251214 02:59:53.866778 21028 catalog_manager.cc:5654] T e0e04af2e6054ecabda3186cef55253f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.871246 21028 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 9 to 10, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 10 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:53.877485 21028 catalog_manager.cc:5654] T 434bbcb8330a4ba680c1346b33460471 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:53.874882 21030 catalog_manager.cc:5654] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: config changed from index 7 to 9, VOTER 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) evicted. New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:53.868361 24073 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:53.886778 21028 catalog_manager.cc:5654] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:53.883838 23957 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20251214 02:59:53.889016 24073 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:53.902006 24073 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:53.915552 24142 raft_consensus.cc:2243] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:53.916553 24142 raft_consensus.cc:2272] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Raft consensus is shut down!
W20251214 02:59:53.935904 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56653us
I20251214 02:59:53.934490 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:53.949257 21205 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20251214 02:59:53.956054 24087 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:53.980227 21191 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c57218e1ae40dfbdcec8dd228432ca with delete_type TABLET_DATA_TOMBSTONED (TS 668ad4f30bd649c88a034bbab9f13b20 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42016
I20251214 02:59:53.987635 23946 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:53.989023 23946 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.002099 21029 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 282) took 6115 ms. Trace:
I20251214 02:59:54.002524 21029 rpcz_store.cc:276] 1214 02:59:47.871288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:47.871529 (+ 241us) service_pool.cc:225] Handling call
1214 02:59:53.984859 (+6113330us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5820,"reported_tablets":43}
I20251214 02:59:54.005784 23946 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.007751 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:54.008385 21124 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 8.
I20251214 02:59:54.007664 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.009505 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.009393 21274 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 02:59:53.993103 24142 ts_tablet_manager.cc:1916] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:53.985605 23906 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.040390 21274 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20251214 02:59:54.041239 23906 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.044346 23906 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:54.046911 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:54.047581 21279 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.050670 24066 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:54.051177 24066 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:54.051532 24066 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.037994 24073 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.056267 24073 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.056854 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:54.057569 21131 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.063661 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:54.064299 21274 raft_consensus.cc:3060] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.072315 21274 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 02:59:54.073406 21985 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.074160 24073 raft_consensus.cc:2804] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:54.074682 24073 raft_consensus.cc:697] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:54.075498 24073 consensus_queue.cc:237] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.076184 21131 raft_consensus.cc:2468] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
W20251214 02:59:54.088681 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 73667.8us
I20251214 02:59:54.089108 21027 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 9 to 10, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 10 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:54.095093 21027 catalog_manager.cc:5654] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 8, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 8 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:54.098068 21027 catalog_manager.cc:5654] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:54.107373 21279 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:54.110613 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:54.111284 22012 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.115010 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:54.119201 24100 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:54.119669 24100 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:54.120509 24100 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.122025 22012 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 02:59:54.125885 21131 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "143de581216646f79bece2b9f1259f64"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:54.126332 21131 raft_consensus.cc:606] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:54.134472 21131 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:54.135025 21131 raft_consensus.cc:606] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:54.138898 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:54.139465 21200 raft_consensus.cc:606] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:54.139776 21200 raft_consensus.cc:614] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:54.196388 24150 tablet_replica.cc:333] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: stopping tablet replica
I20251214 02:59:54.197664 24150 raft_consensus.cc:2243] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Raft consensus shutting down.
I20251214 02:59:54.198544 24150 raft_consensus.cc:2272] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Raft consensus is shut down!
I20251214 02:59:54.170003 24066 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.251730 24066 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.291990 24099 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:54.292716 24099 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.297204 24099 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.303138 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:54.303787 22012 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:54.304977 21096 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.305797 21131 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:54.311954 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:54.312127 23906 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.312527 21126 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.312664 23906 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.313043 24100 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:54.313460 24100 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:54.313812 24100 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:54.316541 23906 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.317034 24142 ts_tablet_manager.cc:1929] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20251214 02:59:54.317436 24142 log.cc:1199] T 3fd36aadbf4841c9a26e48e96430c009 P 6e36caefa37d47a5968cffd9e270044f: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-2-root/wals/3fd36aadbf4841c9a26e48e96430c009
I20251214 02:59:54.320014 21131 raft_consensus.cc:606] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:54.324092 24150 ts_tablet_manager.cc:1916] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251214 02:59:54.326925 21126 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 02:59:54.328860 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:54.329326 21275 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.318892 21017 catalog_manager.cc:4985] TS 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525): tablet 3fd36aadbf4841c9a26e48e96430c009 (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:54.335145 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.335983 21274 raft_consensus.cc:2393] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.338460 21027 catalog_manager.cc:5654] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:54.328982 21985 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.358781 24073 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:54.358860 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.359463 24073 raft_consensus.cc:697] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:54.360411 24073 consensus_queue.cc:237] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.366973 24099 raft_consensus.cc:993] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:54.367501 24099 raft_consensus.cc:1081] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:54.359587 21200 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 11.
I20251214 02:59:54.384694 21275 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 02:59:54.439774 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.440456 21099 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:54.443698 21279 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 11: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.440402 24076 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 02:59:54.462936 24076 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.463810 21205 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:54.464488 21205 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:54.464816 21205 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.465159 24076 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.451848 24099 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 02:59:54.477039 24099 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.477447 24099 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:54.453882 24145 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.480760 24145 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.500808 21205 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.507026 23963 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.507633 23963 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.507768 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:54.508548 21131 raft_consensus.cc:3055] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:54.508860 21131 raft_consensus.cc:740] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:54.509552 21131 consensus_queue.cc:260] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.503718 21205 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.509562 23963 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.512755 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.513378 21203 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 02:59:54.519776 21131 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.531356 23907 raft_consensus.cc:993] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:54.531879 23907 raft_consensus.cc:1081] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:54.538538 21099 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:54.553544 24087 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:54.558601 21029 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 5, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 5 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:54.530107 21131 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:54.568507 21173 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:54.575202 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.582329 23982 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.582880 23982 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.584729 23982 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.584545 24100 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.585815 24088 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:54.586387 24088 raft_consensus.cc:697] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:54.587383 24088 consensus_queue.cc:237] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.563179 24087 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.590931 24087 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:54.591192 23906 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.591719 23906 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.593530 23906 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.566541 24145 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.597422 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.598119 21275 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.602010 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:54.602890 22015 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:54.608950 21205 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "143de581216646f79bece2b9f1259f64"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:54.609580 21205 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:54.609963 21205 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:54.611578 24099 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.613472 24099 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.615105 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.615738 21200 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 02:59:54.614400 21245 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.616694 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
W20251214 02:59:54.624070 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 173447us
I20251214 02:59:54.637907 21205 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.642390 21030 catalog_manager.cc:5654] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 5, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:54.610224 24087 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.673111 21204 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 11.
I20251214 02:59:54.661478 22015 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:54.657768 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:54.625910 24095 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:54.632063 24145 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.691061 21205 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.651199 21200 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:46066
I20251214 02:59:54.691946 21200 raft_consensus.cc:606] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:54.692296 21200 raft_consensus.cc:614] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:54.678900 24095 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:54.695230 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:54.695807 21131 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 02:59:54.696090 21131 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:54.669646 24156 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 02:59:54.701623 21131 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.708810 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:54.710587 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:54.710980 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.711205 22012 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.711671 21274 raft_consensus.cc:2393] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.710558 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:54.691501 24075 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.693573 24095 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:54.725235 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:54.726934 22012 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:54.704502 24100 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:54.714730 23963 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.698539 24145 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.720155 23956 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:54.751034 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:54.729888 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:54.707605 24156 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.724565 24075 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:54.751405 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 119175us
W20251214 02:59:54.795809 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 102314us
I20251214 02:59:54.901057 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.901750 21205 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 02:59:54.902856 21986 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 02:59:54.847325 24162 log.cc:927] Time spent T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20251214 02:59:54.903791 24073 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 02:59:54.904175 24073 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:54.904510 24073 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:54.795675 24076 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.905534 24076 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.846545 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:54.891157 24165 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.909288 24165 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.911865 24073 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.914229 24073 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.914516 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:54.915095 21205 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 7
W20251214 02:59:54.857409 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 141928us
W20251214 02:59:54.765442 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 107776us
I20251214 02:59:54.915072 22016 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:54.909606 24156 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.733361 24070 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251214 02:59:54.928761 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:54.929010 22013 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:54.944170 24164 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.922629 24099 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20251214 02:59:54.977460 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.949352 24095 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:54.950477 24172 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.855633 24145 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.856602 24087 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.911862 24165 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:54.878094 24157 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:54.989795 21124 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:54.973979 21205 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 02:59:54.990227 24157 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.991762 21986 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.747962 23963 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.993212 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.994243 23963 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:54.978969 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:54.996817 21279 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 10: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.978153 21280 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 02:59:54.982080 21096 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:54.938205 24070 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:54.985306 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:54.997737 24070 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:54.998111 21274 raft_consensus.cc:2393] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 11: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:54.985654 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 02:59:54.969882 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 238506us
I20251214 02:59:55.000413 23963 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 02:59:55.010651 24098 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:55.011130 24098 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.755324 23956 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:54.983577 22011 raft_consensus.cc:2368] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:55.012825 23946 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:55.013332 23946 raft_consensus.cc:697] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 02:59:55.013574 23956 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:54.982239 24164 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.014501 23946 consensus_queue.cc:237] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:55.015981 24164 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.024542 21131 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 02:59:54.983318 24076 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:55.032565 22016 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:55.034835 21029 catalog_manager.cc:5654] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 02:59:55.039300 21273 raft_consensus.cc:2393] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:55.041594 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:55.042155 21273 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:55.010362 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.050580 21207 raft_consensus.cc:2410] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20251214 02:59:55.051689 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:55.052340 21276 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 02:59:54.992511 24157 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.009245 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.009608 21274 raft_consensus.cc:1275] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:55.009828 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:55.009956 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:55.065692 21205 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:55.006511 24070 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.058888 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.070184 21277 raft_consensus.cc:2368] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:55.070537 21203 raft_consensus.cc:2393] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:55.059574 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:54.980232 24158 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:55.011181 23963 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:55.071591 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.071172 21200 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 02:59:55.071584 21131 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:55.090893 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.091629 21203 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 02:59:55.092113 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:55.092716 21281 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 02:59:54.986900 24172 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:55.094669 24172 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:55.095278 21279 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 02:59:55.097555 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.098179 21127 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 02:59:55.098868 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.099576 21128 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 02:59:55.100930 21131 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:55.104437 24152 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 02:59:54.987646 24075 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:55.107069 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.109503 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.114610 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 02:59:55.115828 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.116528 21200 raft_consensus.cc:2393] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 02:59:55.117235 23982 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 02:59:55.117686 23982 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:55.118017 23982 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
W20251214 02:59:55.122272 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:55.122773 23956 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 02:59:55.123221 23956 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:55.123631 23956 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:55.124831 21205 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 02:59:55.123075 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 02:59:55.131089 24159 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
W20251214 02:59:55.142825 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 137795us
I20251214 02:59:55.148622 21099 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:55.156437 21124 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:55.164332 21099 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:55.167160 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:55.154032 24099 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 02:59:55.170183 24178 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 02:59:55.170621 24178 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:55.170962 24178 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 02:59:55.172412 23956 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.175303 24098 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.175663 24099 raft_consensus.cc:697] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:55.176254 24095 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:55.175912 23963 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:55.165278 21099 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:55.183537 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 78417.7us
W20251214 02:59:55.184609 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 105821us
I20251214 02:59:55.184840 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.185755 21281 raft_consensus.cc:1275] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251214 02:59:55.186470 21205 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 02:59:55.187896 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.106056 24158 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.193324 21282 raft_consensus.cc:2393] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 02:59:55.194235 24158 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.190308 24164 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 02:59:55.203567 24164 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:55.188261 24156 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 02:59:55.203939 24164 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:55.203150 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:55.204680 22011 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:55.191402 24070 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:55.205955 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.206542 21200 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:55.206912 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.191890 23906 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:55.176770 24099 consensus_queue.cc:237] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.208006 23906 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:55.208884 23906 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.201267 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.210024 24156 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:55.201495 23982 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.201967 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.211329 21130 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 02:59:55.210994 24156 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:55.214742 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
W20251214 02:59:55.217389 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 52189.1us
I20251214 02:59:55.202248 24150 ts_tablet_manager.cc:1929] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.8
I20251214 02:59:55.215950 23982 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.189604 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:55.218822 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.217041 24095 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 02:59:55.288976 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.236964 21171 leader_election.cc:400] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:55.295814 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.298621 21171 leader_election.cc:403] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251214 02:59:55.266800 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.255370 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.298743 21207 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 2.
I20251214 02:59:55.299255 21129 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 2.
I20251214 02:59:55.274819 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.300076 21204 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:55.295291 21200 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 9
W20251214 02:59:55.301580 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 112058us
I20251214 02:59:55.232095 21030 catalog_manager.cc:5654] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251214 02:59:55.308244 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 121228us
I20251214 02:59:55.316521 21204 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:55.317935 21248 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.274426 24159 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:55.320925 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.322397 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.323477 24098 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:55.323982 24098 raft_consensus.cc:697] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:55.324792 24098 consensus_queue.cc:237] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.325502 24156 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7.
I20251214 02:59:55.264873 23956 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.231079 24150 log.cc:1199] T a3c57218e1ae40dfbdcec8dd228432ca P 668ad4f30bd649c88a034bbab9f13b20: Deleting WAL directory at /tmp/dist-test-taskfIxC9K/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1765681046489641-19981-0/minicluster-data/ts-1-root/wals/a3c57218e1ae40dfbdcec8dd228432ca
I20251214 02:59:55.330897 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.344957 24170 raft_consensus.cc:2764] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251214 02:59:55.322434 24183 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20251214 02:59:55.341351 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:55.336475 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.347085 21247 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:55.276814 22011 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:55.328667 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.345721 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:55.388866 24180 log.cc:927] Time spent T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20251214 02:59:55.317572 23946 consensus_queue.cc:1048] T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20251214 02:59:55.298311 24095 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 02:59:55.411803 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 93530.8us
I20251214 02:59:55.414654 21245 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:55.418836 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:55.421307 24166 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 02:59:55.423374 24174 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251214 02:59:55.427273 21079 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 668ad4f30bd649c88a034bbab9f13b20 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:55.390457 21129 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
W20251214 02:59:55.393412 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 48853.9us
I20251214 02:59:55.400234 21127 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 02:59:55.399716 21282 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 1.
W20251214 02:59:55.401908 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 54949.5us
I20251214 02:59:55.429987 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:55.404240 21099 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:55.408627 21204 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 02:59:55.267869 24178 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.413718 24095 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:55.435436 24178 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:55.435940 21128 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:55.435981 24165 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:55.437072 24165 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:55.437947 24165 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:55.440290 21127 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:55.443185 21247 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:55.449051 21128 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:55.454013 24087 raft_consensus.cc:2764] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20251214 02:59:55.456750 24145 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:55.456786 21098 leader_election.cc:400] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:55.449811 24070 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 02:59:55.458050 21200 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 02:59:55.461366 24157 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 02:59:55.459426 24149 consensus_queue.cc:1048] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20251214 02:59:55.462404 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.461760 24076 raft_consensus.cc:2764] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251214 02:59:55.458913 24070 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 02:59:55.467886 21248 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
W20251214 02:59:55.461261 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 639e38cdc0a74102bbcd86211abe87a0, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.468519 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 639e38cdc0a74102bbcd86211abe87a0, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.469233 24173 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election won for term 9
I20251214 02:59:55.469723 24173 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:55.468283 24070 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:55.470945 21208 raft_consensus.cc:2368] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
W20251214 02:59:55.485954 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2734ae274c16411d93b2c95d7fb3e00c, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
W20251214 02:59:55.486358 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2734ae274c16411d93b2c95d7fb3e00c, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:55.461786 24157 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 02:59:55.490062 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 139871us
W20251214 02:59:55.497104 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 434bbcb8330a4ba680c1346b33460471, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.497401 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 434bbcb8330a4ba680c1346b33460471, TS 6e36caefa37d47a5968cffd9e270044f -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.499142 24157 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 1 FOLLOWER]: Advancing to term 2
W20251214 02:59:55.524891 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 163703us
W20251214 02:59:55.525756 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 81941.8us
I20251214 02:59:55.526433 22011 raft_consensus.cc:1275] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:55.526881 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 02:59:55.529253 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 51f125bb79074d9fbb6c280fa03dc07e, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.529613 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 51f125bb79074d9fbb6c280fa03dc07e, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.512153 24150 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42016 (request call id 895) took 1527 ms. Trace:
I20251214 02:59:55.490783 24164 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.557314 24189 log.cc:927] Time spent T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20251214 02:59:55.558734 24190 consensus_queue.cc:1048] T 3fd36aadbf4841c9a26e48e96430c009 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:55.470533 24173 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.609752 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 117109us
I20251214 02:59:55.611744 21275 raft_consensus.cc:2368] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 10. Current term is 11.
W20251214 02:59:55.617357 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 90548.8us
I20251214 02:59:55.547758 23906 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:55.619246 21281 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:55.562975 24145 raft_consensus.cc:993] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:55.630533 21099 leader_election.cc:400] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:55.630964 21099 leader_election.cc:403] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251214 02:59:55.636631 21208 raft_consensus.cc:1275] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 02:59:55.653899 24164 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.632635 24163 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20251214 02:59:55.655279 24169 raft_consensus.cc:1081] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:55.667585 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.670150 21208 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:55.661266 24098 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:55.677543 24182 consensus_queue.cc:1048] T 320811ca9d184da38a1197ebfdb7cd0f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20251214 02:59:55.677773 24098 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.651788 23906 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:55.679298 23906 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.680213 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.633977 24099 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049) with higher term. Message: Invalid argument: T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5.
I20251214 02:59:55.543027 24150 rpcz_store.cc:276] 1214 02:59:53.971142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:53.979885 (+ 8743us) service_pool.cc:225] Handling call
1214 02:59:54.154327 (+174442us) ts_tablet_manager.cc:1151] Deleting tablet a3c57218e1ae40dfbdcec8dd228432ca
1214 02:59:54.158913 (+ 4586us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 02:59:54.591745 (+432832us) tablet_metadata.cc:641] Metadata flushed
1214 02:59:55.202000 (+610255us) tablet_metadata.cc:641] Metadata flushed
1214 02:59:55.488696 (+286696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":163319,"thread_start_us":101952,"threads_started":1}
I20251214 02:59:55.691435 21028 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 3 to 4, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 4 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:55.696990 21029 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 7 to 8, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 8 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:55.682672 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:55.681512 24095 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.719977 21018 catalog_manager.cc:4985] TS 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049): tablet a3c57218e1ae40dfbdcec8dd228432ca (table test-workload [id=89b54530dd4e42b7bc6235879c56c644]) successfully deleted
I20251214 02:59:55.724262 21208 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 11.
W20251214 02:59:55.725647 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 62588.4us
I20251214 02:59:55.727694 24157 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.682271 24098 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.731894 21279 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:55.731954 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:55.732558 21279 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:55.732925 21279 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:55.738042 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:55.731417 24157 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.744973 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:55.746400 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.746747 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.747298 21208 raft_consensus.cc:2393] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 2: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 02:59:55.747288 21204 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:55.744833 24095 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.752729 24070 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.757736 21279 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:55.789108 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 52521.2us
I20251214 02:59:55.793114 21030 catalog_manager.cc:5654] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 5, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:55.790030 21279 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:55.799125 21204 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
W20251214 02:59:55.802608 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 65886.1us
I20251214 02:59:55.788033 24157 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election won for term 11
I20251214 02:59:55.815708 24070 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:55.819303 22011 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:55.844367 21098 leader_election.cc:400] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Vote denied by peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525) with higher term. Message: Invalid argument: T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for earlier term 10. Current term is 11.
I20251214 02:59:55.826313 24157 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:55.861125 24157 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:55.908946 21204 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20251214 02:59:55.910075 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 65873.4us
W20251214 02:59:55.911880 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 167001us
I20251214 02:59:55.923609 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.924170 21204 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:55.925691 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:55.926254 21200 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:55.937615 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.943737 22013 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 02:59:55.943791 21128 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 02:59:55.966612 21204 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:55.955943 24159 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election won for term 10
I20251214 02:59:55.981148 21200 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 02:59:55.982468 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:55.992548 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.993323 21124 raft_consensus.cc:2368] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:59:55.993605 22013 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:55.996359 21278 raft_consensus.cc:1275] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20251214 02:59:55.997270 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:55.997864 21128 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:55.998163 21128 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:55.991163 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:56.002647 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:56.003262 21277 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Advancing to term 3
I20251214 02:59:55.976102 24098 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20251214 02:59:55.984265 24159 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:55.998857 21128 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.004305 21125 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:55.968874 24088 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20251214 02:59:56.076754 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 02:59:56.077577 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 101530us
I20251214 02:59:56.000599 24095 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election won for term 10
W20251214 02:59:56.084352 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 93302.4us
W20251214 02:59:56.048473 24207 log.cc:927] Time spent T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20251214 02:59:56.084107 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 79442.1us
W20251214 02:59:56.084414 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 97347.7us
I20251214 02:59:56.086741 21200 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 02:59:56.089320 21128 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
W20251214 02:59:56.089910 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 91740.6us
I20251214 02:59:56.076596 24165 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 02:59:56.102044 21128 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 02:59:56.109454 21277 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 02:59:56.087081 24169 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20251214 02:59:56.087244 24190 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20251214 02:59:56.122543 24095 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:56.123566 24095 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.115037 24159 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.125972 24210 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20251214 02:59:56.135800 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 51071.3us
I20251214 02:59:56.140203 21278 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: []
W20251214 02:59:56.140597 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 45217.1us
I20251214 02:59:56.141752 24087 consensus_queue.cc:1048] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:56.143651 21098 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:56.145084 24070 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election won for term 3
I20251214 02:59:56.145587 24070 raft_consensus.cc:697] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:56.146430 24070 consensus_queue.cc:237] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.150249 21204 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:56.161757 21247 leader_election.cc:400] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:59:56.162086 21247 leader_election.cc:403] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251214 02:59:56.167615 24210 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855) with higher term. Message: Invalid argument: T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 2. Current term is 3.
I20251214 02:59:56.167771 24178 consensus_queue.cc:1059] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:56.143642 24145 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20251214 02:59:56.170545 23956 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251214 02:59:56.170878 24145 raft_consensus.cc:3055] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:56.179447 24178 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 02:59:56.179839 24178 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:56.180717 21205 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 10. Current term is 11. Ops: []
I20251214 02:59:56.180572 24178 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.171043 24168 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20251214 02:59:56.192049 21127 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251214 02:59:56.194938 24204 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:56.198407 24168 consensus_queue.cc:1243] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer 6e36caefa37d47a5968cffd9e270044f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20251214 02:59:56.181684 24178 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 02:59:56.175145 24145 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 02:59:56.276908 24216 log.cc:927] Time spent T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20251214 02:59:56.290884 24211 log.cc:927] Time spent T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.202s user 0.000s sys 0.001s
I20251214 02:59:56.215777 24222 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20251214 02:59:56.297246 24212 log.cc:927] Time spent T e1dc028308e3434791fdd7169a0858b0 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.203s user 0.003s sys 0.000s
W20251214 02:59:56.363857 21098 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:56.396384 21278 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:56.413741 22016 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20251214 02:59:56.422667 24167 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 02:59:56.476565 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:56.437979 21028 catalog_manager.cc:5654] T 5bdfefe7cfb64f5cba716a427e83fc9f P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 10 to 11, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 11 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:56.503589 21028 catalog_manager.cc:5654] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:56.505137 21028 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 9 to 10, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 10 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:56.528743 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 84747.8us
W20251214 02:59:56.536023 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 86069.9us
I20251214 02:59:56.593873 24145 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:56.595413 24145 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:56.629680 21128 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20251214 02:59:56.630024 21128 pending_rounds.cc:85] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 6
I20251214 02:59:56.630251 21128 pending_rounds.cc:107] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20251214 02:59:56.630468 21128 raft_consensus.cc:2889] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:56.631311 21282 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 10. Current term is 11. Ops: []
I20251214 02:59:56.633459 24198 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Stepping down as leader of term 10
I20251214 02:59:56.633929 24198 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:56.634865 24198 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.635931 24198 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
W20251214 02:59:56.643767 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 101584us
W20251214 02:59:56.672659 24226 log.cc:927] Time spent T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.401s user 0.002s sys 0.000s
W20251214 02:59:56.676692 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 137377us
I20251214 02:59:56.680013 21282 raft_consensus.cc:1275] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:56.680087 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:56.680680 21128 raft_consensus.cc:606] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:56.686216 24166 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:56.687824 21206 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20251214 02:59:56.691134 21096 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 02:59:56.691840 21274 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 9. Current term is 10. Ops: []
I20251214 02:59:56.697860 21203 raft_consensus.cc:1275] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251214 02:59:56.713133 21128 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5751534b61ed4568abd62057238d5fda"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:56.713773 21128 raft_consensus.cc:606] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:56.715617 24164 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:56.717767 24166 consensus_queue.cc:1048] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:56.734419 21129 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:56.734658 23907 consensus_queue.cc:1048] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:56.735028 21129 raft_consensus.cc:606] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:56.740866 21129 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:56.741472 21129 raft_consensus.cc:606] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:56.744586 21129 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:56.745100 21129 raft_consensus.cc:606] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:56.746259 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:56.768435 21203 raft_consensus.cc:1275] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251214 02:59:56.770741 24169 consensus_queue.cc:1048] T 755b818717824383985748e28244d7a1 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251214 02:59:56.795212 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:56.796011 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 02:59:56.800109 21027 catalog_manager.cc:5654] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 4 to 5, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:56.807193 21129 raft_consensus.cc:1275] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20251214 02:59:56.807536 21129 pending_rounds.cc:85] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 6
I20251214 02:59:56.807739 21129 pending_rounds.cc:107] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20251214 02:59:56.807953 21129 raft_consensus.cc:2889] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:56.809227 24158 consensus_queue.cc:1048] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 02:59:56.812318 24169 raft_consensus.cc:993] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:56.812716 24169 raft_consensus.cc:1081] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:56.817305 21203 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:56.817816 21203 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:56.818145 21203 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:56.820225 24197 raft_consensus.cc:993] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:56.820837 24200 raft_consensus.cc:1081] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:56.830104 21274 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:56.830693 21274 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:56.831048 21274 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:56.837321 21274 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.838995 21274 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:56.843323 21206 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20251214 02:59:56.848758 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:56.849272 21206 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:56.851568 21203 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:56.866379 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:56.867048 21130 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:56.867367 21130 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:56.868064 21130 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.869164 21130 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:56.876089 21130 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:56.883569 21247 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:56.884282 24166 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:56.884769 24166 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:56.885607 24166 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:56.893734 24210 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251214 02:59:56.894515 21030 catalog_manager.cc:5654] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:56.896356 21203 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:56.896238 21030 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 8 to 10, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 10 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:56.897758 21030 catalog_manager.cc:5654] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 9 to 10, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 10 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:56.915726 21030 catalog_manager.cc:5654] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 10 to 11, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 11 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:56.932639 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:56.933270 21130 raft_consensus.cc:3055] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Stepping down as leader of term 10
I20251214 02:59:56.933562 21130 raft_consensus.cc:740] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:56.934409 21130 consensus_queue.cc:260] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:56.935568 21130 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
W20251214 02:59:56.940131 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 46574.3us
I20251214 02:59:56.941157 21128 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20251214 02:59:56.942525 21129 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20251214 02:59:56.942905 21129 pending_rounds.cc:85] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 7
I20251214 02:59:56.943116 21129 pending_rounds.cc:107] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20251214 02:59:56.943352 21129 raft_consensus.cc:2889] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 02:59:56.944788 21206 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 02:59:56.954200 23907 raft_consensus.cc:993] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:56.954679 23907 raft_consensus.cc:1081] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:56.959685 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:56.960250 21281 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:56.965570 21278 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:56.966110 21278 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:56.966442 21278 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:56.973253 21200 raft_consensus.cc:1275] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20251214 02:59:56.974364 24095 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:56.975674 24095 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:56.980916 21281 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 02:59:56.985746 21130 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 02:59:56.989624 24210 consensus_queue.cc:1048] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:57.024104 21028 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:57.022903 21278 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:57.046090 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:57.046720 21278 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:57.046741 21128 raft_consensus.cc:3055] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:57.051761 21128 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:57.052479 21128 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:57.053578 21128 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:57.079749 21128 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 02:59:57.085945 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:57.072510 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:57.090293 24166 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:57.103407 24166 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:57.104285 24166 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:57.092252 21278 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 02:59:57.098249 22011 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:57.098877 24198 raft_consensus.cc:993] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:57.126811 24177 raft_consensus.cc:1081] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
W20251214 02:59:57.135885 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 46423.7us
I20251214 02:59:57.138182 21281 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5751534b61ed4568abd62057238d5fda"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:57.138741 21281 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:57.139094 21281 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:57.144371 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 53832.2us
I20251214 02:59:57.144976 24145 raft_consensus.cc:993] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:57.148952 21281 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:57.168043 21281 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:57.172139 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 02:59:57.172809 21203 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:57.152863 24099 raft_consensus.cc:1081] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:57.187567 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:57.188198 21128 raft_consensus.cc:3055] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:57.188476 21128 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:57.189133 21128 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:57.190198 21128 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:57.201239 21206 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:57.201820 21206 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:57.202131 21206 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:57.210253 21203 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:57.212282 21128 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:57.213660 21206 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:57.219961 21206 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:57.283095 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:57.283864 22016 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:57.290695 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 317444us
I20251214 02:59:57.295106 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:57.295675 21124 raft_consensus.cc:3055] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:57.295971 21124 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:57.296180 21173 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:57.296967 24179 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 02:59:57.297473 24179 raft_consensus.cc:697] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [term 11 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:57.298280 24179 consensus_queue.cc:237] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:57.296643 21124 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:57.306002 22016 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:57.306746 22012 raft_consensus.cc:1240] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 02:59:57.313019 21124 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:57.378213 24237 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
I20251214 02:59:57.456455 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
W20251214 02:59:57.458257 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 151763us
W20251214 02:59:57.461798 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 156289us
I20251214 02:59:57.487471 24206 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
W20251214 02:59:57.527154 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 64162.7us
W20251214 02:59:57.523834 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 67465.3us
I20251214 02:59:57.527194 21171 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:57.546283 21124 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 02:59:57.589151 24252 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:57.547660 24206 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:57.592294 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 94388.3us
W20251214 02:59:57.598250 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 93896.9us
I20251214 02:59:57.599822 24252 raft_consensus.cc:697] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:57.600730 24252 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 02:59:57.603070 24256 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20251214 02:59:57.615146 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 67678.1us
I20251214 02:59:57.622238 21204 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:57.607149 22011 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
W20251214 02:59:57.698942 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 54378.8us
I20251214 02:59:57.607048 24206 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:57.716259 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 233409us
I20251214 02:59:57.644438 24166 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20251214 02:59:57.903193 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 178250us
W20251214 02:59:57.912938 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 313855us
I20251214 02:59:57.930815 22012 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251214 02:59:57.940131 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 221030us
I20251214 02:59:57.939020 24267 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20251214 02:59:57.965931 21206 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:57.968835 24276 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 02:59:57.969417 21124 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 02:59:57.975432 21126 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 02:59:57.992755 21124 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 02:59:58.007534 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 1431) took 1004 ms. Trace:
I20251214 02:59:58.007997 21129 rpcz_store.cc:276] 1214 02:59:57.002786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:57.002907 (+ 121us) service_pool.cc:225] Handling call
1214 02:59:57.003229 (+ 322us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:57.003566 (+ 337us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:57.003597 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:57.004058 (+ 461us) log.cc:844] Serialized 26 byte log entry
1214 02:59:57.151278 (+147220us) raft_consensus.cc:1645] Marking committed up to 7
1214 02:59:57.151327 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:57.151573 (+ 246us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:58.006052 (+854479us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b2400104a68. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bd449a4 00007f670bd42aab 00007f670bdd19a9 00007f670bdd1635 00007f670bdd0b90 00007f670eb65828 00007f670eb65f88 00007f670eb3d738 00007f670eb38b0a 00007f670eb37f56
1214 02:59:58.007235 (+ 1183us) raft_consensus.cc:1685] finished
1214 02:59:58.007257 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:58.007465 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1014,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":363323648,"thread_start_us":146934,"threads_started":1,"wal-append.queue_time_us":202189}
W20251214 02:59:58.025992 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 53711.2us
W20251214 02:59:58.107299 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 121640us
I20251214 02:59:58.110005 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251214 02:59:58.114785 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 77261.3us
I20251214 02:59:58.023762 24278 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:58.121627 24277 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:58.121740 24210 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 02:59:58.122716 24278 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:58.123661 24250 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.832s user 0.000s sys 0.017s
I20251214 02:59:58.124714 24278 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:58.122728 24260 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20251214 02:59:58.124763 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47078 (request call id 1430) took 1122 ms. Trace:
W20251214 02:59:58.136575 24253 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.614s user 0.003s sys 0.002s
I20251214 02:59:58.135818 21130 rpcz_store.cc:276] 1214 02:59:57.002478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:57.006201 (+ 3723us) service_pool.cc:225] Handling call
1214 02:59:57.006602 (+ 401us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 02:59:57.006907 (+ 305us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 02:59:57.006934 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 02:59:57.007364 (+ 430us) log.cc:844] Serialized 26 byte log entry
1214 02:59:57.126454 (+119090us) raft_consensus.cc:1645] Marking committed up to 7
1214 02:59:57.126491 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 02:59:57.126557 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 02:59:58.123394 (+996837us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b2400104a68. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bd449a4 00007f670bd42aab 00007f670bdd19a9 00007f670bdd1635 00007f670bdd0c7d 00007f670eb214ae 00007f670eb38c2c 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1
1214 02:59:58.124462 (+ 1068us) raft_consensus.cc:1685] finished
1214 02:59:58.124486 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 02:59:58.124689 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17448,"raft.queue_time_us":15762,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":1624,"spinlock_wait_cycles":366091392,"thread_start_us":118738,"threads_started":1,"wal-append.queue_time_us":159851}
I20251214 02:59:58.142024 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:58.150916 21124 raft_consensus.cc:1275] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20251214 02:59:58.157967 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:58.195890 21247 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:58.211834 24292 consensus_queue.cc:1048] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20251214 02:59:58.243571 24294 log.cc:927] Time spent T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.118s user 0.000s sys 0.002s
I20251214 02:59:58.212307 24164 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:58.348672 24300 log.cc:927] Time spent T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.226s user 0.001s sys 0.002s
I20251214 02:59:58.350070 21028 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 4 to 5, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 5 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251214 02:59:58.354925 24299 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
W20251214 02:59:58.392490 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 211619us
W20251214 02:59:58.498066 24301 log.cc:927] Time spent T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20251214 02:59:58.523501 24198 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:58.528321 21130 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:58.528920 21130 raft_consensus.cc:606] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:58.524693 24198 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 02:59:58.533957 24198 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:58.574800 24222 raft_consensus.cc:2955] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:58.580494 24192 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 02:59:58.581025 24192 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:58.583005 24192 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 02:59:58.575950 21245 consensus_peers.cc:597] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f -> Peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395): Couldn't send request to peer 8de0cf3ca4bf4083bc494a9af0e49342. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c25ad2d7ede4ddeb75d46946114529b. This is attempt 1: this message will repeat every 5th retry.
W20251214 02:59:58.584147 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 58245.5us
W20251214 02:59:58.590168 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 61323.3us
I20251214 02:59:58.593421 22016 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:58.596028 21131 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 02:59:58.598870 21205 raft_consensus.cc:2955] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:58.601118 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 02:59:58.609942 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 02:59:58.615561 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 54843.4us
I20251214 02:59:58.616734 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:58.637693 24178 raft_consensus.cc:993] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:58.638202 24178 raft_consensus.cc:1081] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 02:59:58.641944 21130 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:58.643326 21131 raft_consensus.cc:2955] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Committing config change with OpId 8.6: config changed from index -1 to 6, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:58.622714 24283 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 02:59:58.676456 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:58.678802 21274 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:58.679386 21274 raft_consensus.cc:606] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:58.626426 24254 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20251214 02:59:58.714299 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 85255.4us
I20251214 02:59:58.680485 24283 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:58.699303 24264 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
W20251214 02:59:58.761168 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 141831us
I20251214 02:59:58.769078 21275 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 02:59:58.769609 21275 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:58.769946 21275 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
W20251214 02:59:58.770494 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 143738us
W20251214 02:59:58.774557 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 149871us
I20251214 02:59:58.778796 21275 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:58.779165 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:58.780516 21275 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 02:59:58.780588 23907 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 02:59:58.785967 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:58.800434 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 02:59:58.801055 22012 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:58.807540 24275 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:58.810968 22012 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:58.819294 24283 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 02:59:58.827908 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 144552us
I20251214 02:59:58.838392 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 02:59:58.840376 24222 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:50128 (request call id 6) took 1792 ms. Trace:
I20251214 02:59:58.846714 24260 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 02:59:58.847366 24260 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:58.848287 24260 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:58.858639 24222 rpcz_store.cc:276] 1214 02:59:57.047551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:57.047708 (+ 157us) service_pool.cc:225] Handling call
1214 02:59:57.091380 (+ 43672us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5400233820. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef13a4d 00007f670eefa1d4 00007f670ef7c177 00007f670ef9938a 00007f670ef9a3ea 00007f670ef9959b 00007f670efa4cbc 00007f670efa4333 00007f6713f54859 00007f670d4e8334
1214 02:59:57.109828 (+ 18448us) log.cc:844] Serialized 506 byte log entry
1214 02:59:57.986296 (+876468us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b180012a780. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6714d41f21 00007f6714d4e901 00007f670beeffe3 00007f670bef23b1 00007f670bef2369 00007f670bef22f9 00007f670bef22c1 00007f670bef15bd 0000000000379254 0000000000379099 00007f670bed33aa
1214 02:59:58.135939 (+149643us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 02:59:58.272096 (+136157us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b180012a780. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6714d41f21 00007f6714d4e901 00007f670bef0112 00007f670bef23b1 00007f670bef2369 00007f670bef22f9 00007f670bef22c1 00007f670bef15bd 0000000000379254 0000000000379099 00007f670bed33aa
1214 02:59:58.273444 (+ 1348us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b58005e01d0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef080c0 00007f670eef846c 00007f670eefed83 00007f670eefed09 00007f670eefec99 00007f670eefec61 00007f670eefde2d 0000000000379254 0000000000379099 00007f670bef0065 00007f670bef23b1
1214 02:59:58.452471 (+179027us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400233898. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef5a4ec 00007f670ef5cff0 00007f670ef5cf78 00007f670ef5cef8 00007f670ef5ceb3 00007f670ef5c17c 00007f6714dd1cc6 00007f6714dcc478 00007f670eb44ee1 00007f670eb382ca
1214 02:59:58.840286 (+387815us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8236,"raft.queue_time_us":133168,"raft.run_cpu_time_us":2832,"raft.run_wall_time_us":381579,"spinlock_wait_cycles":438945024,"thread_start_us":339710,"threads_started":3,"wal-append.queue_time_us":365935}
W20251214 02:59:58.861362 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 60986.3us
W20251214 02:59:58.864957 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 79084us
I20251214 02:59:58.865958 24158 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:58.871447 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:58.872004 21124 raft_consensus.cc:606] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 02:59:58.872313 21203 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 02:59:58.872324 21124 raft_consensus.cc:614] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 02:59:58.876037 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:58.878958 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:58.879495 21129 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 02:59:58.879758 21129 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:58.880402 21129 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:58.881565 21129 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 02:59:58.884378 21275 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 02:59:58.884927 21275 raft_consensus.cc:606] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:58.887761 21124 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 02:59:58.888331 21124 raft_consensus.cc:606] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 02:59:58.929826 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251214 02:59:58.931761 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 02:59:58.972148 24315 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 02:59:58.975423 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:58.987952 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:59.002846 24283 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 02:59:59.009857 21129 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 02:59:59.026513 24266 raft_consensus.cc:993] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.029268 21029 catalog_manager.cc:5654] T 2734ae274c16411d93b2c95d7fb3e00c P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 10 to 11, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 11 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } }
I20251214 02:59:59.036005 21029 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 6 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:59.063565 24328 ts_tablet_manager.cc:933] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Initiating tablet copy from peer 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:59.131661 24319 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 02:59:59.140738 24319 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 02:59:59.142704 24319 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 02:59:59.150197 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 02:59:59.151062 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 02:59:59.155406 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 02:59:59.156596 24272 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 02:59:59.182139 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 172396us
W20251214 02:59:59.181696 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 172623us
I20251214 02:59:59.192298 24324 raft_consensus.cc:1081] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.198760 24276 raft_consensus.cc:993] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.199240 24276 raft_consensus.cc:1081] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.228349 21202 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:59.228937 21202 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:59.229312 21202 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:59.230779 21204 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 02:59:59.231333 21204 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:59.231654 21204 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:59.233865 24295 raft_consensus.cc:993] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.234344 24295 raft_consensus.cc:1081] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 02:59:59.258734 22016 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 02:59:59.303776 21204 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.323427 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251214 02:59:59.271682 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 45137.5us
W20251214 02:59:59.314142 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 46300.5us
I20251214 02:59:59.337805 21202 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.377436 21205 raft_consensus.cc:1240] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 10. Current term is 11. Ops: []
I20251214 02:59:59.382126 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:59.382851 21131 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:59.385076 24316 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:59.407446 21204 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:59.407506 21124 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 02:59:59.410313 21199 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42048
I20251214 02:59:59.410882 21199 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 02:59:59.411232 21199 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:59.416697 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:59.417304 21277 raft_consensus.cc:3055] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Stepping down as leader of term 4
I20251214 02:59:59.417572 21277 raft_consensus.cc:740] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 02:59:59.420506 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:59.421034 21130 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:59.418973 21131 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 02:59:59.422935 21202 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:59.423380 21199 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.426632 21173 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:59.427047 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:59.427583 21275 raft_consensus.cc:3055] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Stepping down as leader of term 10
I20251214 02:59:59.427860 21275 raft_consensus.cc:740] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
W20251214 02:59:59.430671 21079 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251214 02:59:59.431699 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 02:59:59.428566 21275 consensus_queue.cc:260] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.435395 21275 raft_consensus.cc:3060] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 02:59:59.440608 24321 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 02:59:59.328580 24328 tablet_copy_client.cc:323] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Beginning tablet copy session from remote peer at address 127.19.131.67:34525
I20251214 02:59:59.444407 21291 tablet_copy_service.cc:140] P 6e36caefa37d47a5968cffd9e270044f: Received BeginTabletCopySession request for tablet 0c25ad2d7ede4ddeb75d46946114529b from peer 8de0cf3ca4bf4083bc494a9af0e49342 ({username='slave'} at 127.0.0.1:50140)
I20251214 02:59:59.444953 21291 tablet_copy_service.cc:161] P 6e36caefa37d47a5968cffd9e270044f: Beginning new tablet copy session on tablet 0c25ad2d7ede4ddeb75d46946114529b from peer 8de0cf3ca4bf4083bc494a9af0e49342 at {username='slave'} at 127.0.0.1:50140: session id = 8de0cf3ca4bf4083bc494a9af0e49342-0c25ad2d7ede4ddeb75d46946114529b
I20251214 02:59:59.418254 21277 consensus_queue.cc:260] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.447583 21277 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 02:59:59.451915 21130 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:59.461536 21199 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 02:59:59.462476 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 02:59:59.463025 21274 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:59.466264 24264 raft_consensus.cc:2804] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 02:59:59.467012 24264 raft_consensus.cc:697] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [term 11 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:59.467833 24264 consensus_queue.cc:237] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.472460 21277 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 02:59:59.466292 21173 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 02:59:59.474911 21275 raft_consensus.cc:2468] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 02:59:59.475750 21274 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:59.474929 24254 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 02:59:59.478328 21173 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 02:59:59.479460 24254 raft_consensus.cc:697] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 02:59:59.470335 21291 tablet_copy_source_session.cc:215] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: Tablet Copy: opened 0 blocks and 1 log segments
I20251214 02:59:59.480327 24254 consensus_queue.cc:237] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.487653 24292 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 02:59:59.488194 24292 raft_consensus.cc:697] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 02:59:59.488543 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2ce7181915c14c369bac4b962d34913a, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:59.488940 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2ce7181915c14c369bac4b962d34913a, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.489172 24292 consensus_queue.cc:237] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:59.502344 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5751534b61ed4568abd62057238d5fda, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:59.502821 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5751534b61ed4568abd62057238d5fda, TS 668ad4f30bd649c88a034bbab9f13b20 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.542855 21131 raft_consensus.cc:3055] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 02:59:59.547014 21131 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 02:59:59.547801 21131 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:59.546393 21079 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 51f125bb79074d9fbb6c280fa03dc07e, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 02:59:59.549187 21079 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 51f125bb79074d9fbb6c280fa03dc07e, TS 721d604130924d939d291fe3eef8f6d1 -> TS 8de0cf3ca4bf4083bc494a9af0e49342 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 02:59:59.552878 21131 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 02:59:59.559561 21131 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 02:59:59.563563 21205 raft_consensus.cc:1240] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 4. Current term is 5. Ops: []
I20251214 02:59:59.576184 24328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c25ad2d7ede4ddeb75d46946114529b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 02:59:59.589139 21030 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 170) took 1016 ms. Trace:
I20251214 02:59:59.589829 21030 rpcz_store.cc:276] 1214 02:59:58.572747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:58.572869 (+ 122us) service_pool.cc:225] Handling call
1214 02:59:59.589064 (+1016195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":741,"reported_tablets":3}
I20251214 02:59:59.596293 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51662 (request call id 310) took 1858 ms. Trace:
I20251214 02:59:59.601166 21027 catalog_manager.cc:5654] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } }
I20251214 02:59:59.604333 21027 catalog_manager.cc:5654] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 reported cstate change: config changed from index -1 to 6, NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) added. New cstate: current_term: 8 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } } }
I20251214 02:59:59.596693 21028 rpcz_store.cc:276] 1214 02:59:57.728498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 02:59:57.728656 (+ 158us) service_pool.cc:225] Handling call
1214 02:59:59.586426 (+1857770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 02:59:58.602116 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 02:59:58.602654 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
1214 02:59:58.602683 (+ 29us) write_op.cc:435] Acquired schema lock
1214 02:59:58.602699 (+ 16us) tablet.cc:660] Decoding operations
1214 02:59:58.603004 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
1214 02:59:58.603078 (+ 74us) write_op.cc:641] Partition lock acquired for write op
1214 02:59:58.603104 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
1214 02:59:58.603288 (+ 184us) tablet.cc:699] Row locks acquired
1214 02:59:58.603306 (+ 18us) write_op.cc:260] PREPARE: finished
1214 02:59:58.603439 (+ 133us) write_op.cc:270] Start()
1214 02:59:58.603535 (+ 96us) write_op.cc:276] Timestamp: P: 1765681198603394 usec, L: 0
1214 02:59:58.603554 (+ 19us) op_driver.cc:348] REPLICATION: starting
1214 02:59:58.604133 (+ 579us) log.cc:844] Serialized 536 byte log entry
1214 02:59:58.888311 (+284178us) op_driver.cc:464] REPLICATION: finished
1214 02:59:59.481303 (+592992us) write_op.cc:301] APPLY: starting
1214 02:59:59.481378 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
1214 02:59:59.481466 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
1214 02:59:59.481587 (+ 121us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 02:59:59.481771 (+ 184us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 02:59:59.481927 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 02:59:59.481978 (+ 51us) write_op.cc:312] APPLY: finished
1214 02:59:59.555647 (+ 73669us) log.cc:844] Serialized 23 byte log entry
1214 02:59:59.584231 (+ 28584us) write_op.cc:489] Releasing partition, row and schema locks
1214 02:59:59.584315 (+ 84us) write_op.cc:454] Released schema lock
1214 02:59:59.584356 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":586358,"num_ops":1,"prepare.queue_time_us":80613,"prepare.run_cpu_time_us":24526,"prepare.run_wall_time_us":64854,"raft.queue_time_us":66283,"raft.run_cpu_time_us":45804,"raft.run_wall_time_us":458401,"replication_time_us":284576,"thread_start_us":476090,"threads_started":4,"wal-append.queue_time_us":111374}]]}
I20251214 02:59:59.648036 21030 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 02:59:59.685470 21028 catalog_manager.cc:5654] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 10 to 11, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 11 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:59.687031 21028 catalog_manager.cc:5654] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 4 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:59.688501 21028 catalog_manager.cc:5654] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 6 to 7, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 7 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } }
I20251214 02:59:59.700790 24328 tablet_copy_client.cc:806] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 0 data blocks...
I20251214 02:59:59.701390 24328 tablet_copy_client.cc:670] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Starting download of 1 WAL segments...
I20251214 02:59:59.706845 24328 tablet_copy_client.cc:538] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251214 02:59:59.911512 21131 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:59.913038 21275 raft_consensus.cc:1275] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251214 02:59:59.916261 24254 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 02:59:59.930696 24328 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap starting.
I20251214 02:59:59.940526 24297 consensus_queue.cc:1048] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 03:00:00.003141 21128 raft_consensus.cc:1275] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 03:00:00.013507 24292 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251214 03:00:00.026893 21277 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20251214 03:00:00.035285 24282 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251214 03:00:00.054069 21128 raft_consensus.cc:1275] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20251214 03:00:00.058838 24282 consensus_queue.cc:1048] T 2ce7181915c14c369bac4b962d34913a P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251214 03:00:00.104054 21278 raft_consensus.cc:1275] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251214 03:00:00.111407 24292 consensus_queue.cc:1048] T 8c3be94c517348a08049e54abf8fd780 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20251214 03:00:00.359604 24328 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 03:00:00.360584 24328 tablet_bootstrap.cc:492] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Bootstrap complete.
I20251214 03:00:00.361266 24328 ts_tablet_manager.cc:1403] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent bootstrapping tablet: real 0.434s user 0.045s sys 0.047s
I20251214 03:00:00.394025 24328 raft_consensus.cc:359] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:00.403735 24328 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Initialized, Role: LEARNER
I20251214 03:00:00.404523 24328 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:00.438748 24328 ts_tablet_manager.cc:1434] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent starting tablet: real 0.077s user 0.009s sys 0.001s
I20251214 03:00:00.450344 21291 tablet_copy_service.cc:342] P 6e36caefa37d47a5968cffd9e270044f: Request end of tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-0c25ad2d7ede4ddeb75d46946114529b received from {username='slave'} at 127.0.0.1:50140
I20251214 03:00:00.450980 21291 tablet_copy_service.cc:434] P 6e36caefa37d47a5968cffd9e270044f: ending tablet copy session 8de0cf3ca4bf4083bc494a9af0e49342-0c25ad2d7ede4ddeb75d46946114529b on tablet 0c25ad2d7ede4ddeb75d46946114529b with peer 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:00.570780 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 03:00:00.899848 22011 raft_consensus.cc:1217] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 LEARNER]: Deduplicated request from leader. Original: 8.5->[8.6-8.6] Dedup: 8.6->[]
I20251214 03:00:01.634282 24354 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.635016 24354 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:01.637302 24354 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:01.643391 24359 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.643968 24359 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:01.646085 24359 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.661832 24358 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:01.662542 24358 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:01.938885 24358 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:01.939456 24358 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.939980 24358 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:01.941958 24358 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.732482 24360 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.943202 24360 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:01.945175 24360 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.771003 24361 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:01.946692 24361 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:01.948621 24361 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:01.954500 24359 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:01.955117 24359 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:01.957402 24359 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.957966 24359 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:01.958039 24364 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.958475 24359 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:01.960520 24359 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.961009 24359 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:01.961500 24359 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:01.963465 24359 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:01.964010 24359 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:01.962275 24365 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:01.861649 21124 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 584) took 1028 ms. Trace:
I20251214 03:00:01.910302 24367 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.836568 24363 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.968119 24359 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:01.819375 24362 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.056175 24354 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:01.924738 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:01.973506 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.525619 24366 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.526355 24366 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.408509 21127 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 03:00:02.548604 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.251078 21124 rpcz_store.cc:276] 1214 03:00:00.832973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:01.860583 (+1027610us) service_pool.cc:225] Handling call
1214 03:00:01.860985 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:01.861186 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:01.861214 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:01.861255 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:01.861283 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:01.861336 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:01.861462 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 03:00:02.547443 24362 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:02.554443 21229 debug-util.cc:398] Leaking SignalData structure 0x7b080025c920 after lost signal to thread 19984
I20251214 03:00:02.055815 24364 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.186774 24365 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.568177 24370 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:02.568852 24370 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.570704 24371 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:02.547570 24363 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:02.571476 21096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.003s sys 0.014s
I20251214 03:00:02.571303 24371 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:02.571867 21096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.004s sys 0.014s
I20251214 03:00:02.572739 24363 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.547224 24354 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.573518 24366 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.573935 24359 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.574100 24366 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.571025 24370 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.575101 24366 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.575783 24364 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.491710 24367 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.576303 24364 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:02.576733 24364 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.577095 24366 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.577636 24366 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:02.571218 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 217463us
I20251214 03:00:02.574397 24354 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.578150 24366 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.598745 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:02.584348 24366 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.578528 24364 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.596524 24360 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.654662 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:02.596457 24358 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:02.654670 24369 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.586097 24361 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.638602 24370 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.584213 24292 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.641397 24368 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.637835 24363 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.658090 24375 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.652719 24371 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:02.579665 24362 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.659403 22013 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:02.580027 24354 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.577800 24367 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.658949 24360 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:02.779466 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.003s sys 0.011s
I20251214 03:00:02.582661 24314 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.584025 24365 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.775282 24369 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:02.780292 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.003s sys 0.012s
I20251214 03:00:02.776487 24370 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.775411 24358 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.777987 24292 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.792721 24375 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.793732 24370 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.794266 24370 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.794440 24292 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.794777 24375 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.777143 24361 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.804173 24361 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.804705 24370 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.779899 24362 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.798242 24358 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.792207 24376 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:02.807518 24362 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.778304 24363 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.798632 24369 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.802354 24360 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.779057 24371 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.780592 24367 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.814095 24364 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.779743 24354 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.779013 24368 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.833175 24377 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 03:00:02.834722 21172 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.017s sys 0.078s
W20251214 03:00:02.835634 21172 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.018s sys 0.078s
I20251214 03:00:02.830847 24358 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.826915 24375 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:02.836162 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.018s sys 0.046s
I20251214 03:00:02.825953 24376 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 03:00:02.834254 24364 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.837726 24292 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.830157 24367 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.839339 24292 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.841068 24367 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.841244 24292 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.841681 24358 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.843662 24368 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.843782 24358 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.845907 24376 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:02.828658 24371 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.839212 24360 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.840428 24364 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.848050 24360 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.838186 24377 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.850013 24360 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.850313 24377 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.850579 24375 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.851179 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.847496 24374 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:02.851824 24374 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.853336 24362 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.847566 24354 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:02.842067 24370 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.855086 24365 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:02.855707 24375 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.855592 24365 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:02.839303 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.853917 24374 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.857951 21127 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 03:00:02.861419 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.868465 21124 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 03:00:02.813351 24314 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:02.997380 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:02.943034 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 73407.7us
I20251214 03:00:02.891218 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.870123 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.180188 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 583) took 2347 ms. Trace:
I20251214 03:00:03.187913 21124 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:02.997824 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:02.915910 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:02.997584 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.188052 21130 rpcz_store.cc:276] 1214 03:00:00.832315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:03.179171 (+2346856us) service_pool.cc:225] Handling call
1214 03:00:03.179559 (+ 388us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:03.179749 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:03.179775 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:03.179813 (+ 38us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:03.179847 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:03.179898 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:03.180019 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 03:00:03.005023 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.011413 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.241503 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.238131 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.218080 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.193697 21129 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 03:00:03.091406 24098 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.005427 24269 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.189399 21199 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 11.
I20251214 03:00:02.866994 24364 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:02.866962 24359 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:03.190604 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 185657us
I20251214 03:00:02.892024 24354 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.262243 24354 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.264243 24354 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:02.938889 24363 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.264882 24315 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.265391 24315 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.266839 24166 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.267549 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 174) took 2300 ms. Trace:
I20251214 03:00:03.267340 24166 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:03.269462 21229 debug-util.cc:398] Leaking SignalData structure 0x7b0800297840 after lost signal to thread 21065
I20251214 03:00:03.270367 24166 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.270915 24166 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.271373 24166 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.264909 24359 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.274698 24364 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.275480 24359 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.265408 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.276608 24364 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.276901 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.277172 24364 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:03.277535 24364 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.277863 24364 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.273067 24315 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.005285 24374 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:02.894749 24375 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:02.870663 24282 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.252854 21204 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 03:00:02.870750 24370 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:02.935173 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:02.870312 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.176s user 0.028s sys 0.027s
I20251214 03:00:03.249074 22016 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
W20251214 03:00:03.301460 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.607s user 0.034s sys 0.064s
I20251214 03:00:03.260303 24264 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 03:00:03.249012 24280 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.261528 24357 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.250064 22012 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:03.238936 24379 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.246022 21096 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.267900 21027 rpcz_store.cc:276] 1214 03:00:00.966661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:00.966770 (+ 109us) service_pool.cc:225] Handling call
1214 03:00:03.267375 (+2300605us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20251214 03:00:03.190375 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.279687 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.033147 24371 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.089062 24285 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.279668 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:02.870867 24362 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.248853 21200 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:03.303069 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.299468 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 03:00:03.303891 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 57960.5us
I20251214 03:00:03.138633 24381 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.281774 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.303391 21199 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:02.891309 24373 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.308589 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.317948 21204 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:03.310011 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.305332 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:03.314038 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.314195 24357 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.315027 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.314858 24370 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.325217 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.279045 24315 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.308717 24379 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.300506 24366 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:03.316012 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.328675 21202 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:03.308868 24367 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:03.320601 21200 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 3.
I20251214 03:00:03.322244 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.300364 24382 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.329077 21096 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.328884 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.329221 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.324923 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.331576 21099 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.331802 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.339833 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.335197 22012 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:03.341467 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.335620 21204 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 03:00:03.342818 21099 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.334422 21133 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 03:00:03.315969 21171 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.68:33395 timed out after 1.596s (SENT) [suppressed 4 similar messages]
I20251214 03:00:03.253783 24098 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.386430 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.379899 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:03.380424 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 51445.4us
I20251214 03:00:03.388739 21096 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.386657 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:03.345548 21229 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0a20 after lost signal to thread 21157
I20251214 03:00:03.325362 24365 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.339948 24315 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.331324 24366 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.386246 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.340778 24367 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:03.392107 21229 debug-util.cc:398] Leaking SignalData structure 0x7b080025c0a0 after lost signal to thread 21232
I20251214 03:00:03.310585 24374 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.305006 24375 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.393991 21208 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:03.343453 24377 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251214 03:00:03.397105 21229 debug-util.cc:398] Leaking SignalData structure 0x7b0800314520 after lost signal to thread 21306
I20251214 03:00:03.390300 24361 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:03.386174 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.380187 24360 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:03.400252 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.317804 24362 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.401304 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.386436 24359 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:03.310459 24269 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:03.415163 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 115783us
I20251214 03:00:03.324116 24282 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.417645 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.418870 24198 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:03.411963 24377 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.323694 24373 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.423470 24370 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.423748 24166 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.323616 24280 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.424299 24395 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.424857 24373 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.395525 24366 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:03.425426 24380 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.425632 24280 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.425830 24379 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.427400 24374 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.428952 24375 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.397768 24367 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:03.424930 24363 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:03.431227 24282 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.433175 24362 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.311993 24285 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.416154 24359 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.434635 24385 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.422921 24098 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.436002 24285 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.419767 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.436731 24198 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.437145 24198 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:03.434866 24359 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:03.431186 24363 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.439076 24363 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:03.439342 24357 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.440095 24377 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:03.440455 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.424546 24264 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.441089 24264 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:03.422091 21205 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:03.430752 21199 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:03.422709 24290 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:03.445763 24269 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.445935 24290 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.413965 24360 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.422958 24314 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.446436 24360 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:03.436209 24387 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.418817 23906 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 03:00:03.449072 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.446358 24290 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:03.447165 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:03.449456 21229 debug-util.cc:398] Leaking SignalData structure 0x7b08000fe6e0 after lost signal to thread 22044
W20251214 03:00:03.451177 21229 debug-util.cc:398] Leaking SignalData structure 0x7b080038cc40 after lost signal to thread 22045
I20251214 03:00:03.441138 21207 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 03:00:03.464905 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.465881 24394 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:03.466028 24393 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.466331 24394 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.466729 24394 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.470446 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 03:00:03.471366 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 82721.9us
W20251214 03:00:03.471822 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 129078us
I20251214 03:00:03.471953 24269 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:03.472396 24269 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.472442 21099 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.472849 24269 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:03.473323 21096 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.474854 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.458551 23906 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.475741 23906 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:03.476272 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.463711 24098 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.477593 24098 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.478703 24384 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.479553 24098 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.480170 24389 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.481586 24391 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.484947 24264 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.449774 21201 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:03.449692 24361 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.487150 21099 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.498664 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:03.498981 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 183042us
I20251214 03:00:03.434896 24380 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.506842 24380 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.379323 24382 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.507794 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.466629 24393 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.480821 24389 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 03:00:03.508725 24382 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.447805 24387 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.508993 24366 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.510133 21208 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:03.510362 24389 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.482122 24391 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.510747 24387 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.512295 24391 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.512847 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.513427 22016 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:03.514488 21096 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.479223 24384 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.500918 24378 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:03.498834 24361 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:03.463949 24385 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.499418 21129 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 03:00:03.502699 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:03.521315 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.502878 21099 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.500169 24359 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.424862 24395 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.503901 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.504420 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.504973 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.321069 24381 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.500629 24370 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:03.580647 24370 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.581089 24370 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.581187 24393 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.520550 24378 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.581950 24378 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.582862 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.584606 23906 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.586784 24389 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.586930 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.587324 24389 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.587757 24359 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.588096 24391 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.508438 21203 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 03:00:03.588603 24391 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.522833 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.449092 24315 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.521497 24392 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.523485 24364 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.575712 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:03.577343 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 74554.7us
I20251214 03:00:03.577813 24360 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:03.579953 24395 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.580158 24166 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.587844 24381 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.618695 24388 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.619455 24360 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.619251 24388 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.620822 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.621289 24388 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.611299 24362 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.532684 24380 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:03.622587 24380 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.617658 21202 raft_consensus.cc:2393] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 4: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:03.623036 24380 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:03.609916 24392 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.624104 21205 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:03.624812 24392 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.622026 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.626240 24392 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.596436 21204 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:03.596858 24391 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.636104 24390 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.579850 24098 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.597244 21207 raft_consensus.cc:1240] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 11. Current term is 12. Ops: []
I20251214 03:00:03.636624 24390 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.597654 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.637403 24098 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.635526 24382 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.639101 24382 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.640420 22016 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:03.640607 24394 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.640969 24382 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.593739 24357 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:03.636214 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.593907 23906 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.600162 24389 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.598287 24366 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.642678 24394 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.600589 24363 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.643817 24390 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.645550 24363 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.602495 24198 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.603379 24404 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.646723 24404 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.646890 24392 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.604063 24280 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.604918 24397 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.605476 24361 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.607239 24385 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.527143 24387 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.607472 24377 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.653640 24387 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.653865 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.522137 24367 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.658221 24367 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.658756 21128 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:03.615180 24166 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.615831 24403 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.621944 24364 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.594120 21200 raft_consensus.cc:2393] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:03.622531 24362 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.627151 24398 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.666343 24403 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.666872 24198 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.627990 24285 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.666388 24362 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.538005 24264 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.508492 24290 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.667595 24285 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.668704 24403 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.669450 24290 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.669601 21206 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:03.630892 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.669955 24285 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.633062 24405 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.632977 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.633540 24395 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.671267 24395 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.672470 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.634848 24380 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.635532 24400 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.639417 24098 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.515662 24401 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:03.676966 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.642023 24357 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.600019 24269 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.671022 24405 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.677883 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.678865 24396 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:03.649519 24280 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.649713 24404 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.650985 24392 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.679322 21203 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:03.679152 24405 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.651914 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.676090 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.680683 24280 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.677300 24401 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.681918 24401 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.650219 24397 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.682595 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.684293 24397 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.654886 24377 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.684609 24166 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.520190 24384 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.655766 24387 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.685304 24395 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.650233 24370 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.652649 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.686743 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.661657 22016 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.664022 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.664659 21124 raft_consensus.cc:1240] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:03.666688 24398 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.688691 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.688967 24396 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.689429 24396 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.689759 24357 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:03.690303 24398 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.627609 24315 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:03.691064 24315 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.692972 24315 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.674494 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.675367 24400 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.677421 24388 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20251214 03:00:03.681452 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.695384 24388 raft_consensus.cc:3055] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 03:00:03.695740 24388 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:03.696575 24388 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.697805 24388 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.695911 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:03.679152 24269 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.699411 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.699713 24404 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:03.700146 22013 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 3.
I20251214 03:00:03.699457 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.701236 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.698031 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.702085 22017 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:03.702904 24269 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:03.702055 24377 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:03.646420 24378 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.697017 24361 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.694743 21130 raft_consensus.cc:1240] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:03.705178 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.704514 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:03.697922 24397 consensus_queue.cc:1175] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "721d604130924d939d291fe3eef8f6d1" responder_term: 3 status { last_received { term: 3 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 5 } }
I20251214 03:00:03.706458 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.705818 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.706194 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.716431 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.716670 24269 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.717165 24269 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.717344 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.717614 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.711578 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.705466 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.713105 24377 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.691540 24380 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.732091 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.732257 24400 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:03.713562 24401 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.746093 24396 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.747366 22011 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:03.734018 24264 consensus_queue.cc:1059] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20251214 03:00:03.751935 24264 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Stepping down as leader of term 8
I20251214 03:00:03.752269 24264 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:03.729387 24377 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.757117 24396 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.758090 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:03.758659 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 71036.5us
I20251214 03:00:03.759513 24400 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:03.752545 21125 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:03.760030 24400 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.761870 24400 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.752179 24378 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.765776 24264 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.767316 24264 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:03.771725 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:03.772795 21124 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:03.775812 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.780278 24357 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.783305 24357 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.797896 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:03.798636 24364 consensus_queue.cc:1059] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:03.720094 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:03.799831 21126 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
I20251214 03:00:03.800021 22014 raft_consensus.cc:1240] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:03.798996 21133 raft_consensus.cc:2393] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:03.610450 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:03.807523 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 197165us
I20251214 03:00:03.827154 24361 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 03:00:03.827394 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.827695 24361 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:03.828323 24361 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:03.828625 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.829521 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.750473 24269 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.725665 24370 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.831045 24394 raft_consensus.cc:3055] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:03.831395 24394 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:03.832413 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.832172 24394 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.833437 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.833967 24269 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.833369 24394 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:03.834550 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.837455 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.838636 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.838848 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:03.839744 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.832188 24377 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.841315 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.848217 24377 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.852427 21986 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:03.852515 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43946.1us
I20251214 03:00:03.854197 24396 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:03.861294 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.862020 21200 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 10.
I20251214 03:00:03.863349 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.866528 21986 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.867853 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.867784 24396 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 03:00:03.868738 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.868685 24396 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.869508 24396 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:03.858304 24406 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20251214 03:00:03.869987 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.870496 24406 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:03.870888 24406 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:03.871378 21099 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.871640 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.871675 24406 consensus_queue.cc:260] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.873786 24406 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:03.875622 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.876713 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:03.877408 22011 raft_consensus.cc:2393] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:03.858455 24401 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.880038 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.880344 21099 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.868335 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.883186 22015 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.886678 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.886897 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.887279 21274 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:03.887481 21275 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 03:00:03.886273 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.888569 21280 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:03.889593 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.889956 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.890165 24370 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:03.890724 24370 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.891173 24370 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:03.879940 24401 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:03.891651 21277 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:03.892786 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.881749 21099 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.894003 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.895226 21099 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:03.892413 21171 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:03.896634 21099 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:03.898553 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.897037 24378 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251214 03:00:03.899595 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.900590 21282 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:03.899822 24378 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.901100 24378 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:03.902077 21171 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:03.896924 24380 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:03.902894 21130 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:03.902895 24380 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.903553 24380 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:03.899811 24269 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:03.905833 24394 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:03.908006 24378 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.903519 21282 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 03:00:03.909098 21282 raft_consensus.cc:606] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:03.892232 24401 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:03.912878 24401 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:03.914095 24396 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.915128 24380 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:03.916643 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.923346 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.923936 24357 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:03.927510 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.920977 22015 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:03.922716 21204 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:03.914271 22011 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 03:00:03.926162 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.929337 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.919540 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:03.931666 21171 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:03.924557 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.932963 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.916484 24361 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.934471 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.962306 21205 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 12: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:03.962954 24396 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 11 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.963124 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.963905 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:03.964157 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.965068 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.965719 24378 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.938707 21207 raft_consensus.cc:1240] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:03.967973 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.968667 21207 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:03.969321 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.939555 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.971918 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.973002 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.973982 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.941242 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.975184 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.975726 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.976313 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.942086 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.941885 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.977408 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.977906 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.978960 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.979926 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.942800 24361 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:03.980938 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.942814 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.982059 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.943495 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.983021 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.983238 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.983979 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.950657 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.952251 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.941318 24357 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:03.956219 24264 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election won for term 4
I20251214 03:00:03.942493 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.987879 24264 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:03.989698 24264 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:03.945219 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.934185 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.939589 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.934190 24367 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:03.939289 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.940116 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:03.976783 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:03.934346 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:03.934263 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.041115 24367 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:03.946367 21204 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 11.
I20251214 03:00:04.049443 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.001561 21206 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:03.953047 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:03.961289 24380 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.940963 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.053889 21986 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:04.054212 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.055004 24396 consensus_queue.cc:1059] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:04.055344 24367 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.055742 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.055197 24388 raft_consensus.cc:2804] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 03:00:03.960165 24401 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.056571 24388 raft_consensus.cc:697] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:04.057178 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.057461 24388 consensus_queue.cc:237] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.058028 24401 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:03.934016 24370 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.058835 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.082666 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.083546 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.058894 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.085695 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.059582 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.059744 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.060288 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.087481 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.056010 24396 raft_consensus.cc:3055] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 03:00:04.061558 24397 consensus_queue.cc:1059] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:04.061931 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.092264 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.060994 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.062453 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.063030 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.063285 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.063483 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.062687 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.064240 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.065267 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.066174 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.066927 24406 consensus_queue.cc:1059] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:04.067926 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.096649 21130 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:04.072939 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.096220 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.073716 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.051990 21205 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:04.066421 21173 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.060923 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.088615 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.091235 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.091444 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.094630 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.068894 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.099985 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.102443 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.102993 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.106216 24393 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:04.114212 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.107868 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.121989 21126 raft_consensus.cc:1240] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:04.123678 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:04.090992 24396 raft_consensus.cc:740] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:04.130507 21124 raft_consensus.cc:1240] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:04.136128 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.304695 24396 consensus_queue.cc:260] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.306253 24396 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:04.306476 21027 catalog_manager.cc:5654] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 4, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 4 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251214 03:00:04.309720 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:04.113531 24282 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 03:00:04.314939 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.142927 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.316378 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.316695 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.317423 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.317826 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.318325 22015 raft_consensus.cc:1240] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:04.319005 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.257329 24407 raft_consensus.cc:3055] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:04.319187 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.330480 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.333818 21133 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:04.333971 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.319823 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.331136 21099 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.323256 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.336573 21099 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.339462 21133 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:04.340128 21124 raft_consensus.cc:1240] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:04.340427 24370 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:04.341048 24370 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:04.341480 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.334743 24407 raft_consensus.cc:740] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:04.339922 21130 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:04.342211 21127 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:04.343905 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.346668 21275 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:04.345218 21099 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.343859 21274 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "721d604130924d939d291fe3eef8f6d1"
from {username='slave'} at 127.0.0.1:43308
I20251214 03:00:04.347744 21274 raft_consensus.cc:606] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Received request to transfer leadership to TS 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.348160 21274 raft_consensus.cc:614] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 03:00:04.349123 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.349472 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.350646 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.349013 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:04.342526 24198 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 03:00:04.332306 24393 consensus_queue.cc:1059] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20251214 03:00:04.359704 24393 raft_consensus.cc:3055] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 03:00:04.360090 24393 raft_consensus.cc:740] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:04.360922 24393 consensus_queue.cc:260] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.365798 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:04.364277 24393 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:04.370066 24407 consensus_queue.cc:260] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.375586 24407 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:04.368335 24406 consensus_queue.cc:1059] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20251214 03:00:04.406401 24400 consensus_queue.cc:1059] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251214 03:00:04.510227 24407 raft_consensus.cc:3055] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:04.510675 24407 raft_consensus.cc:740] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:04.511495 24407 consensus_queue.cc:260] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.512749 24407 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
W20251214 03:00:04.515419 24416 log.cc:927] Time spent T 00000000000000000000000000000000 P 86f60dbd6dbf443abf2563a85669a2b8: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20251214 03:00:04.676710 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 270201us
I20251214 03:00:04.686481 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:04.665024 24412 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.240s user 0.000s sys 0.001s
I20251214 03:00:04.690135 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.691198 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.691797 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.691897 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.692574 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.695989 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.689168 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:04.700099 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.703622 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.705726 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.707927 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:04.711531 21208 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 10. Current term is 11. Ops: []
W20251214 03:00:04.721119 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:04.725382 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:04.731287 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:04.739777 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 394595us
I20251214 03:00:04.743571 24198 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:04.744141 24198 raft_consensus.cc:697] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:04.745093 24198 consensus_queue.cc:237] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.757609 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.759737 24370 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20251214 03:00:04.759884 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 693520us
I20251214 03:00:04.762727 24364 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251214 03:00:04.763738 21099 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.765844 21099 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:04.774693 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 66898.1us
I20251214 03:00:04.776185 21099 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.778895 21985 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:04.779707 24388 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:04.780128 24388 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:04.780478 24388 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:04.779835 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.778321 24357 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 03:00:04.781716 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.772663 24367 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 03:00:04.781872 24357 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:04.783313 24357 raft_consensus.cc:697] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:04.784221 24357 consensus_queue.cc:237] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.788560 21985 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:04.789822 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:04.792588 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:04.801863 24417 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:04.803937 21985 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:04.805666 21985 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 03:00:04.837280 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:04.802179 24397 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:04.804605 24415 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:04.858616 24415 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:04.859093 24415 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:04.859905 24430 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251214 03:00:04.860452 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 54683.4us
I20251214 03:00:04.804679 24417 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:04.863559 21985 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.874733 24417 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:04.876417 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 82048.8us
I20251214 03:00:04.876824 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:04.877559 21131 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:04.851159 24397 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:04.810755 24388 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.882501 22013 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:04.885459 24417 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:04.890065 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:04.867293 24421 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251214 03:00:04.867456 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 87085.1us
I20251214 03:00:04.868388 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.870609 24427 raft_consensus.cc:993] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 03:00:04.838076 24406 consensus_queue.cc:1059] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20251214 03:00:04.866710 21245 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.879734 24397 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:04.862116 24413 consensus_queue.cc:1059] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20251214 03:00:04.877997 24430 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:04.900249 24418 raft_consensus.cc:2749] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:04.900903 24418 raft_consensus.cc:1081] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 03:00:04.901683 24406 raft_consensus.cc:3055] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:04.902096 24406 raft_consensus.cc:740] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:04.902913 24406 consensus_queue.cc:260] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.903470 24413 raft_consensus.cc:3055] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 LEADER]: Stepping down as leader of term 11
I20251214 03:00:04.903853 24413 raft_consensus.cc:740] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:04.904138 24406 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:04.904570 24413 consensus_queue.cc:260] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:04.906984 24414 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:04.895398 24421 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:04.907464 24421 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:04.910774 24428 consensus_queue.cc:1059] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:04.911860 24428 raft_consensus.cc:3055] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:04.912211 24428 raft_consensus.cc:740] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:04.896132 21245 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.893976 21098 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:04.920012 21098 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.920595 21248 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.921895 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.923081 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.925287 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:04.926446 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:04.934324 21098 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:04.935096 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.954896 21245 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 03:00:05.052762 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 239110us
I20251214 03:00:05.064301 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.065814 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:04.935767 24357 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:04.923981 24414 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:04.933495 24198 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:04.936689 24427 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:05.099577 21208 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20251214 03:00:05.054790 24406 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.066835 24419 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:05.054956 24388 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.063349 24424 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:05.090502 24364 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:05.107292 24414 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 03:00:05.110517 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 176241us
I20251214 03:00:05.172473 21098 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:05.109117 24357 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20251214 03:00:05.185381 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:05.188751 21098 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:05.111719 24427 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:05.109889 24429 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:05.216591 24427 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:05.240628 24427 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
W20251214 03:00:05.248309 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 182580us
I20251214 03:00:05.102099 24428 consensus_queue.cc:260] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:05.261198 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:05.190675 24417 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:05.288050 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 333247us
W20251214 03:00:05.143291 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 208258us
I20251214 03:00:05.190507 24406 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:05.243441 24357 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.191402 24415 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.325932 24429 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:05.325546 24426 raft_consensus.cc:2749] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:05.326184 21248 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.327277 21245 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:05.327780 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:05.315480 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 126803us
I20251214 03:00:05.242425 24436 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 03:00:05.355446 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:05.346905 24428 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:05.259356 24397 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.385763 24421 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.349380 24440 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:05.386896 24440 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:05.388470 24440 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:05.315259 24437 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.143s
I20251214 03:00:05.355556 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
W20251214 03:00:05.671864 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 316400us
I20251214 03:00:05.673221 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.355614 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.356464 21245 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:05.682772 21098 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:05.683782 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.027s sys 0.020s
I20251214 03:00:05.684900 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:05.685577 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.028s sys 0.020s
I20251214 03:00:05.351148 24442 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:05.345664 24406 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:05.696344 21983 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.013s sys 0.056s
W20251214 03:00:05.716053 21983 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.013s sys 0.056s
I20251214 03:00:05.691751 24419 raft_consensus.cc:2749] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:05.697124 24442 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:05.700990 24198 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.702312 24439 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251214 03:00:05.688393 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 332044us
I20251214 03:00:05.356974 24414 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:05.357190 24418 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:05.685673 24401 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:05.709479 24443 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:05.744311 24419 consensus_queue.cc:1059] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:05.745402 24419 raft_consensus.cc:3055] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Stepping down as leader of term 3
I20251214 03:00:05.745755 24419 raft_consensus.cc:740] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:05.712697 24415 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.357107 24436 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.747808 24406 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:05.699146 24445 raft_consensus.cc:2749] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:05.748636 24436 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:05.749511 24442 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.746472 24419 consensus_queue.cc:260] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.749989 24439 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:05.750427 24439 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:05.687613 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 332064us
I20251214 03:00:05.752574 21248 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.753788 21248 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.754374 24406 raft_consensus.cc:2749] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:05.754739 24419 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:05.756353 24397 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.736083 24434 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:05.759058 24434 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:05.761039 24434 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.763391 24401 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:05.687314 21173 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:05.764743 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 77503.5us
I20251214 03:00:05.765470 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.766438 24442 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:05.766724 21245 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.767189 24442 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 03:00:05.772210 24427 consensus_meta.cc:287] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.001s
I20251214 03:00:05.772631 24440 consensus_queue.cc:1059] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:05.785369 24440 raft_consensus.cc:3055] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:05.785789 24440 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:05.786609 24440 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.787851 24440 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:05.789292 24370 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:05.797668 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:05.807664 24439 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.765717 24421 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:05.780042 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:05.780545 21245 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.781477 24418 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:05.782346 24414 raft_consensus.cc:1064] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f: attempting to promote NON_VOTER 8de0cf3ca4bf4083bc494a9af0e49342 to VOTER
I20251214 03:00:05.824710 24414 consensus_queue.cc:237] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 4, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.827585 24439 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:05.784123 21248 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:05.835290 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 51238.1us
I20251214 03:00:05.835994 21248 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.837016 21248 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.808624 21173 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.837805 24444 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:05.775431 24429 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:05.819962 24427 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:05.821537 21171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.046s sys 0.085s
I20251214 03:00:05.838699 24419 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:05.829602 24418 raft_consensus.cc:2749] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:05.839475 24418 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:05.839416 21248 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.838171 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.840497 24427 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:05.841068 24427 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.840395 24418 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20251214 03:00:05.841663 24444 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:05.841405 21173 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.843294 21173 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.844072 24364 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251214 03:00:05.845144 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:05.846149 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:05.848238 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:05.848973 21201 raft_consensus.cc:2393] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:05.849452 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:05.850080 24264 raft_consensus.cc:2749] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251214 03:00:05.849011 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 166291us
I20251214 03:00:05.844743 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.850773 22013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44282 (request call id 860) took 1077 ms. Trace:
I20251214 03:00:05.851178 22013 rpcz_store.cc:276] 1214 03:00:04.773571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:04.773724 (+ 153us) service_pool.cc:225] Handling call
1214 03:00:04.782862 (+ 9138us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:04.882879 (+100017us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:05.850509 (+967630us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54004d0dd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 03:00:05.850694 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228771072}
I20251214 03:00:05.851590 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:05.851974 21248 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.853214 21248 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:05.854079 24418 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:05.854929 24418 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.855468 24418 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:05.783537 24428 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 03:00:05.830155 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 49702.7us
I20251214 03:00:05.856758 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 299) took 1553 ms. Trace:
I20251214 03:00:05.856906 21245 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.857630 24428 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251214 03:00:05.834082 21985 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.041s sys 0.070s
W20251214 03:00:05.858392 21985 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.041s sys 0.070s
I20251214 03:00:05.859014 21245 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:05.859690 24428 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:05.789857 24370 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:05.839017 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s user 0.046s sys 0.085s
I20251214 03:00:05.860716 21248 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:05.861218 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:05.861975 24428 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:05.851074 21098 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:05.862910 24282 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:05.863951 24282 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:05.870514 24282 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:05.868793 24449 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:05.857178 21027 rpcz_store.cc:276] 1214 03:00:04.303348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:04.303501 (+ 153us) service_pool.cc:225] Handling call
1214 03:00:05.856666 (+1553165us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 03:00:04.384078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 03:00:04.384696 (+ 618us) write_op.cc:432] Acquiring schema lock in shared mode
1214 03:00:04.384749 (+ 53us) write_op.cc:435] Acquired schema lock
1214 03:00:04.384783 (+ 34us) tablet.cc:660] Decoding operations
1214 03:00:04.385197 (+ 414us) write_op.cc:620] Acquiring the partition lock for write op
1214 03:00:04.385294 (+ 97us) write_op.cc:641] Partition lock acquired for write op
1214 03:00:04.385323 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
1214 03:00:04.385585 (+ 262us) tablet.cc:699] Row locks acquired
1214 03:00:04.385620 (+ 35us) write_op.cc:260] PREPARE: finished
1214 03:00:04.385814 (+ 194us) write_op.cc:270] Start()
1214 03:00:04.385994 (+ 180us) write_op.cc:276] Timestamp: P: 1765681204385741 usec, L: 0
1214 03:00:04.386040 (+ 46us) op_driver.cc:348] REPLICATION: starting
1214 03:00:04.386839 (+ 799us) log.cc:844] Serialized 536 byte log entry
1214 03:00:04.853188 (+466349us) op_driver.cc:464] REPLICATION: finished
1214 03:00:04.938856 (+ 85668us) write_op.cc:301] APPLY: starting
1214 03:00:04.938948 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
1214 03:00:04.939064 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
1214 03:00:04.939212 (+ 148us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 03:00:04.939496 (+ 284us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 03:00:04.939718 (+ 222us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 03:00:04.939762 (+ 44us) write_op.cc:312] APPLY: finished
1214 03:00:04.940028 (+ 266us) log.cc:844] Serialized 23 byte log entry
1214 03:00:04.940238 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:04.940322 (+ 84us) write_op.cc:454] Released schema lock
1214 03:00:04.940373 (+ 51us) write_op.cc:341] FINISH: Updating metrics
1214 03:00:04.940764 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:04.940800 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":85409,"apply.run_cpu_time_us":1893,"apply.run_wall_time_us":1894,"num_ops":1,"prepare.queue_time_us":72472,"prepare.run_cpu_time_us":15103,"prepare.run_wall_time_us":40994,"raft.queue_time_us":336329,"raft.run_cpu_time_us":11552,"raft.run_wall_time_us":62818,"replication_time_us":466968,"thread_start_us":784783,"threads_started":4,"wal-append.queue_time_us":38291}]]}
W20251214 03:00:05.837301 21984 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.029s sys 0.091s
W20251214 03:00:06.607316 21984 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.416s user 0.065s sys 0.157s
I20251214 03:00:06.608891 24264 consensus_queue.cc:1059] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:06.610013 24396 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:06.610628 24396 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.612488 24396 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.614060 24397 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.614527 24397 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.616367 24397 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.616797 24397 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:06.617199 24397 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.622164 24396 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:06.624269 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.624899 21207 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 03:00:06.626113 21986 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:06.622766 24396 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.629681 24396 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.630270 24396 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:06.630668 24396 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:06.083254 24449 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.184863 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 605) took 2103 ms. Trace:
I20251214 03:00:06.636646 24396 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.636679 24449 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.639001 24449 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:06.187273 24450 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.639628 24450 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.641557 24450 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:06.239346 24444 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:06.521850 24420 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 03:00:06.523557 24364 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:06.643283 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.643601 24364 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:06.636664 21132 rpcz_store.cc:276] 1214 03:00:04.081884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:04.124556 (+ 42672us) service_pool.cc:225] Handling call
1214 03:00:06.184110 (+2059554us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b5800120998. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670beb2ac1 00007f670beb2972 00007f6714d5b069 00007f6713edd260 00007f6713fa5dbb 00007f6713fa5e41 00007f6713f67af0 00007f6713f52459 00007f6713f51f02 00007f670d4ded34
1214 03:00:06.184330 (+ 220us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:06.184520 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:06.184544 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:06.184585 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:06.184620 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:06.184675 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:06.184784 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12504832}
I20251214 03:00:06.645431 24364 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:06.524850 21200 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.014s sys 0.016s
I20251214 03:00:06.645521 24444 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:06.646068 21200 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s user 0.015s sys 0.016s
I20251214 03:00:06.553082 24282 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.647259 24282 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.566675 22014 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:06.649150 24282 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:06.566584 24280 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:06.649755 24280 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.651721 24280 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.569110 24430 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.567797 24448 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:06.569890 24436 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.652858 24430 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.653218 24436 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.571803 24421 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 03:00:06.572289 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 713313us
I20251214 03:00:06.569507 24415 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.655253 21245 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.655174 24415 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:06.573367 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 712697us
I20251214 03:00:06.581074 24370 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.567698 24437 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:06.657182 24448 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.657534 24437 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.659132 24448 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.659471 24437 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:05.845121 21173 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:06.660452 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 815398us
I20251214 03:00:06.660889 24436 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.600246 24400 raft_consensus.cc:3055] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:06.661253 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:06.661568 24364 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 03:00:06.662045 22017 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 4.
I20251214 03:00:06.663076 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:06.643245 24420 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:06.664443 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:06.664573 24420 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.665102 22017 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 03:00:06.654829 24421 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.666332 21171 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:06.667168 24364 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 03:00:06.667536 24364 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:06.667644 24421 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.667919 24364 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:06.668262 24413 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:06.661378 24400 raft_consensus.cc:740] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:06.665686 24407 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:06.670068 24400 consensus_queue.cc:260] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:06.671016 24407 consensus_queue.cc:1059] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20251214 03:00:06.671286 24400 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.662308 22016 raft_consensus.cc:1240] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:06.661129 21173 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:06.673393 24450 consensus_queue.cc:1059] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:06.673880 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.674381 24450 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:06.675098 24450 raft_consensus.cc:3055] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 03:00:06.675411 24450 raft_consensus.cc:740] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:06.676153 24450 consensus_queue.cc:260] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.676519 24364 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.632644 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.634114 22015 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:06.677541 21207 raft_consensus.cc:2393] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:06.677459 24450 raft_consensus.cc:3060] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
W20251214 03:00:06.679324 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:06.679864 24364 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:06.680406 24364 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:06.680490 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:06.681092 22015 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:06.665616 24420 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:06.656984 24415 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.654812 24430 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.583146 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.687134 21206 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:06.689983 21099 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:06.690793 24370 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:06.691175 24370 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:06.691533 24370 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:06.695618 21096 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:06.696270 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.696522 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.696976 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.697697 21202 raft_consensus.cc:2393] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 5: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:06.698630 24400 raft_consensus.cc:3055] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Stepping down as leader of term 10
I20251214 03:00:06.699003 24400 raft_consensus.cc:740] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:06.697639 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.691438 22013 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:06.701224 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.701794 21200 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 03:00:06.703318 22014 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20251214 03:00:05.895673 21098 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.647547 24444 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:06.676911 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:06.705453 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:06.635986 22019 raft_consensus.cc:1240] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:06.631345 24397 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.706699 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:06.714339 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 841647us
I20251214 03:00:06.716390 21098 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:06.717213 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:06.718554 21098 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:06.719797 22018 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:06.642112 24454 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.711856 22015 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
W20251214 03:00:06.714090 21281 consensus_meta.cc:287] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.864s user 0.007s sys 0.010s
I20251214 03:00:06.726404 21281 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:06.720929 21173 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:06.726778 24367 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251214 03:00:06.727133 24460 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:06.727659 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:06.728602 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:06.727800 24460 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:06.733466 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 78287.5us
I20251214 03:00:06.700337 24380 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251214 03:00:06.734261 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 77339.5us
I20251214 03:00:06.715277 24401 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:06.739022 21245 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.720118 24377 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:06.681053 24364 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20251214 03:00:06.725380 24282 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:06.729063 24361 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251214 03:00:06.729386 24421 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 03:00:06.748981 21203 raft_consensus.cc:1275] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20251214 03:00:06.749840 24407 consensus_queue.cc:1059] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 03:00:06.725802 24444 consensus_queue.cc:1059] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20251214 03:00:06.740363 24380 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:06.740361 24429 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:06.765964 21201 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 03:00:06.754041 24459 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:06.775202 24430 consensus_queue.cc:1059] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20251214 03:00:06.781437 24198 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:06.782495 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:06.755553 24421 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:06.735970 24454 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.788205 24413 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:06.791864 24394 raft_consensus.cc:3055] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:06.792596 21199 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 10.
I20251214 03:00:06.771036 24380 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:06.753150 24282 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:06.796772 22013 raft_consensus.cc:1275] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20251214 03:00:06.797147 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 65939.1us
W20251214 03:00:06.794404 21281 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 3144) took 2626 ms (client timeout 2202 ms). Trace:
W20251214 03:00:06.807150 21281 rpcz_store.cc:269] 1214 03:00:04.158483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:05.797423 (+1638940us) service_pool.cc:225] Handling call
1214 03:00:06.784717 (+987294us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54000483d4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 03:00:06.784828 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118113664}
I20251214 03:00:06.739038 24400 consensus_queue.cc:260] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:06.798305 21276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 3138) took 2719 ms (client timeout 2196 ms). Trace:
I20251214 03:00:06.803037 21027 catalog_manager.cc:5654] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 reported cstate change: term changed from 10 to 11, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 11 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 03:00:06.798030 24421 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:06.772423 24449 consensus_queue.cc:1048] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
W20251214 03:00:06.810106 21276 rpcz_store.cc:269] 1214 03:00:04.079235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:06.798121 (+2718886us) service_pool.cc:212] Skipping call since client already timed out
1214 03:00:06.798235 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251214 03:00:06.812844 22012 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 03:00:06.799537 24264 raft_consensus.cc:2749] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:06.774744 24429 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.806533 24444 raft_consensus.cc:3055] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:06.803542 24394 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
W20251214 03:00:06.826629 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 99041.8us
I20251214 03:00:06.801981 24450 consensus_queue.cc:1059] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20251214 03:00:06.819303 24413 raft_consensus.cc:3055] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:06.821655 24400 raft_consensus.cc:3060] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:06.828413 21173 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.829892 24377 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:06.835836 24377 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:06.837930 24377 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:06.832238 24413 raft_consensus.cc:740] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:06.838882 24394 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.839030 24413 consensus_queue.cc:260] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:06.839810 24429 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:06.840140 24394 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.796064 24459 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:06.841986 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.842339 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:06.815248 24401 consensus_queue.cc:1048] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20251214 03:00:06.826697 24444 raft_consensus.cc:740] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:06.848819 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 03:00:06.850203 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 117013us
I20251214 03:00:06.871258 21099 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.840142 24413 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
W20251214 03:00:06.874485 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 135559us
W20251214 03:00:06.874752 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 46435.4us
I20251214 03:00:06.828876 24396 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.876274 21245 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:06.878373 21173 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:06.881822 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:06.882048 21171 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:06.845976 24370 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:06.895861 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800321580 after lost signal to thread 19984
I20251214 03:00:06.897569 24459 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 03:00:06.897735 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08002ce4c0 after lost signal to thread 21065
W20251214 03:00:06.900564 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08002405a0 after lost signal to thread 21157
I20251214 03:00:06.901136 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:06.901269 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:06.901893 22014 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:06.902079 22015 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 LEARNER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20251214 03:00:06.902310 21274 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251214 03:00:06.902663 21154 debug-util.cc:398] Leaking SignalData structure 0x7b0800321380 after lost signal to thread 21232
I20251214 03:00:06.903789 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:06.876698 24429 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:06.878733 24465 raft_consensus.cc:3055] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
W20251214 03:00:06.903707 21154 debug-util.cc:398] Leaking SignalData structure 0x7b080035e320 after lost signal to thread 21306
I20251214 03:00:06.891551 24282 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
W20251214 03:00:06.917156 21154 debug-util.cc:398] Leaking SignalData structure 0x7b08002b2360 after lost signal to thread 22044
I20251214 03:00:06.894840 24436 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:06.929124 22013 raft_consensus.cc:1240] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 10. Current term is 11. Ops: []
I20251214 03:00:06.928323 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:06.850090 24463 consensus_queue.cc:1059] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20251214 03:00:06.940845 22011 raft_consensus.cc:2393] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:06.908097 24264 raft_consensus.cc:2804] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 03:00:06.948977 24264 raft_consensus.cc:697] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [term 5 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:06.923044 24364 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20251214 03:00:06.952953 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 61715.1us
I20251214 03:00:06.898144 24459 consensus_queue.cc:1059] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20251214 03:00:06.955020 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 216407us
I20251214 03:00:06.914937 24465 raft_consensus.cc:740] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:06.939446 24436 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:06.963320 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.963987 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.964570 21126 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:06.964679 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.965206 21130 raft_consensus.cc:2393] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:06.965849 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.966068 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.966605 21132 raft_consensus.cc:2393] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 03:00:06.967269 21098 connection.cc:232] RPC call timeout handler was delayed by -1.5355363380000426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
I20251214 03:00:06.967447 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:06.967722 21098 connection.cc:232] RPC call timeout handler was delayed by -1.3585787529999607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20251214 03:00:06.968075 21098 connection.cc:232] RPC call timeout handler was delayed by -1.3248917199999823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20251214 03:00:06.968474 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:06.968341 21098 connection.cc:232] RPC call timeout handler was delayed by -1.3198361629999908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
I20251214 03:00:06.969158 21133 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 10: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 03:00:06.969357 21098 connection.cc:232] RPC call timeout handler was delayed by -1.132803478000028s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
I20251214 03:00:06.969485 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:06.969681 21098 connection.cc:232] RPC call timeout handler was delayed by -1.1267086439999048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2147s for a response
W20251214 03:00:06.969992 21098 connection.cc:232] RPC call timeout handler was delayed by -1.0901480389999278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17390000000000003s for a response
I20251214 03:00:06.906136 24444 consensus_queue.cc:260] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.973078 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.966415 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.974151 21985 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:06.974879 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.976734 24370 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:06.977046 24466 log.cc:927] Time spent T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20251214 03:00:06.977735 24460 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.986366 22012 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:06.989722 24396 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:06.991194 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:06.991761 21205 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.995110 24264 consensus_queue.cc:237] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:06.969858 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.980167 21132 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
W20251214 03:00:06.976622 24469 log.cc:927] Time spent T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20251214 03:00:07.019199 21133 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:39374
I20251214 03:00:07.019933 21133 raft_consensus.cc:606] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Received request to transfer leadership to TS 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:07.022104 21124 raft_consensus.cc:1240] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:06.982287 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:06.983547 24444 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.984092 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:06.984851 24465 consensus_queue.cc:260] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.002342 21280 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:07.012446 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:07.040122 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 84291.7us
I20251214 03:00:07.014835 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:07.041404 21132 raft_consensus.cc:3060] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:06.960633 24436 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:07.016074 24364 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.043372 24364 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.044107 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.044904 24465 raft_consensus.cc:3060] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.022884 21207 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.045307 24364 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:07.024191 21125 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 10. Current term is 11. Ops: []
I20251214 03:00:07.024447 21131 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:07.024730 21128 raft_consensus.cc:1240] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:07.025455 21130 raft_consensus.cc:1240] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:06.976960 24415 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:07.045176 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.048262 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.049263 21131 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:07.052495 21130 raft_consensus.cc:1240] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:07.039563 21129 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:07.055138 21128 raft_consensus.cc:1240] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: []
W20251214 03:00:07.052625 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 57709.8us
I20251214 03:00:07.059525 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.060091 21274 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:06.998684 24430 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:07.114951 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.115222 21203 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:07.119093 21275 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:07.118366 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.119136 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.084270 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.123361 21277 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.122619 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.124505 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.118834 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:07.121572 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 56160.3us
I20251214 03:00:07.125167 21204 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:07.066108 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.115307 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.084770 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.061117 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.131006 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.131735 21273 raft_consensus.cc:2393] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 4: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:07.132696 21204 raft_consensus.cc:1275] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20251214 03:00:07.132970 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.133260 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.133912 21127 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:07.063697 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 03:00:07.139108 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:07.134666 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 256340us
I20251214 03:00:07.083181 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.065084 24396 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251214 03:00:07.147604 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 93155.3us
I20251214 03:00:07.145509 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.052953 24380 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:07.152446 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 298181us
I20251214 03:00:07.152711 21205 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:07.145143 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:06.960250 24467 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:07.151446 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.148242 24463 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:07.141031 21199 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b"
dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:42032
I20251214 03:00:07.017091 24472 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:07.143918 21173 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.147486 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.065578 24421 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.061442 24454 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:07.143815 21279 raft_consensus.cc:2393] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:07.156652 21199 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20251214 03:00:07.146234 24482 raft_consensus.cc:3055] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Stepping down as leader of term 8
I20251214 03:00:07.155290 21273 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 3.
I20251214 03:00:07.158183 21280 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 03:00:07.166152 21199 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:07.167021 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 155013us
I20251214 03:00:07.156133 21275 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:07.157106 21986 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:07.135926 21278 raft_consensus.cc:3060] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:07.078552 24459 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.206406 21126 raft_consensus.cc:2393] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 5: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.209496 21173 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.166486 24472 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.155234 24485 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 03:00:07.267179 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 58738.6us
W20251214 03:00:07.046077 24455 consensus_queue.cc:1175] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" responder_term: 7 status { last_received { term: 6 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251214 03:00:07.060905 24477 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20251214 03:00:07.269044 21132 raft_consensus.cc:2468] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
W20251214 03:00:07.270048 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.269346 21199 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:07.211206 24415 raft_consensus.cc:2804] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election won for term 6
W20251214 03:00:07.271266 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 61810us
I20251214 03:00:07.282258 21278 raft_consensus.cc:2468] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:07.284286 21203 raft_consensus.cc:2468] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 03:00:07.160706 24463 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.316295 24463 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.167687 24482 raft_consensus.cc:740] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.204346 24478 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:07.288919 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.205829 24264 raft_consensus.cc:2749] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251214 03:00:07.322853 21173 connection.cc:232] RPC call timeout handler was delayed by -1.5114140150000139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20251214 03:00:07.309525 21199 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:07.286114 24485 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.313992 21276 raft_consensus.cc:1275] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20251214 03:00:07.314332 21282 raft_consensus.cc:2393] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 03:00:07.318938 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:07.326805 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.284858 24459 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.329772 24459 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.330605 24482 consensus_queue.cc:260] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.282949 24364 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 03:00:07.337807 21277 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 03:00:07.338058 21207 raft_consensus.cc:2468] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:07.202893 24462 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20251214 03:00:07.340466 21206 raft_consensus.cc:1240] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: []
W20251214 03:00:07.347707 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 266520us
I20251214 03:00:07.350630 22011 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:07.351397 21099 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 03:00:07.351980 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.339223 24280 raft_consensus.cc:3055] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Stepping down as leader of term 4
W20251214 03:00:07.353226 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.341022 24420 raft_consensus.cc:3055] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Stepping down as leader of term 6
W20251214 03:00:07.354676 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.364545 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.362423 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:07.365919 22011 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
W20251214 03:00:07.367242 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 231022us
I20251214 03:00:07.402050 22011 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:07.418972 21096 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.412070 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:07.421778 22014 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:07.423746 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 03:00:07.424943 21172 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.426265 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.426602 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.427130 21132 raft_consensus.cc:2393] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.428361 21127 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10. Ops: []
I20251214 03:00:07.429922 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.431514 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.432287 21127 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 4.
I20251214 03:00:07.354398 24280 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:07.352662 24415 raft_consensus.cc:697] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:07.433722 21126 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 03:00:07.434042 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 158819us
I20251214 03:00:07.434492 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:07.380143 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.334285 24478 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.439924 21248 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:07.440557 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.394394 24472 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:07.447925 21132 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 11: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.358263 24364 consensus_queue.cc:1059] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20251214 03:00:07.442726 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.410516 24493 raft_consensus.cc:2804] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:07.447141 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:07.451921 21985 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.440284 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.452626 21131 raft_consensus.cc:3060] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:07.436015 24413 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:07.435204 24489 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:07.435910 24449 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 03:00:07.436713 24421 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:07.448822 24477 raft_consensus.cc:2804] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader election won for term 4
W20251214 03:00:07.447889 24394 consensus_meta.cc:287] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.607s user 0.008s sys 0.006s
I20251214 03:00:07.457553 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.458004 21124 raft_consensus.cc:1240] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:07.455698 24413 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:07.470537 24413 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:07.471132 24280 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.472381 24280 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Advancing to term 5
W20251214 03:00:07.473156 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.360940 24380 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:07.454171 24483 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:07.463336 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.477469 21985 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.476341 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:07.357162 24420 raft_consensus.cc:740] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:07.478112 24477 raft_consensus.cc:697] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [term 4 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.475056 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.480624 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.487805 21199 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 03:00:07.488040 21277 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:07.481688 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:07.483525 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 419761us
I20251214 03:00:07.456169 24454 raft_consensus.cc:2764] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251214 03:00:07.490465 24420 consensus_queue.cc:260] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.491662 24420 raft_consensus.cc:3060] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.484163 21127 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.484138 21281 raft_consensus.cc:1240] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:07.495221 24485 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:07.470698 24484 log.cc:927] Time spent T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20251214 03:00:07.478965 24483 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:07.495970 24483 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.447955 24463 consensus_queue.cc:1048] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20251214 03:00:07.458540 24493 raft_consensus.cc:697] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:07.499533 21131 raft_consensus.cc:2468] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 03:00:07.499415 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:07.500978 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:07.477689 24449 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:07.511478 24449 consensus_queue.cc:237] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:07.512850 24282 consensus_meta.cc:287] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.570s user 0.006s sys 0.002s
I20251214 03:00:07.513279 24282 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:07.472855 24489 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.520133 21281 raft_consensus.cc:1240] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:07.484722 24437 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 03:00:07.580349 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 79659.7us
I20251214 03:00:07.526630 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:07.558152 21986 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:07.558708 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.577678 24463 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:07.586648 21202 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:07.495810 24478 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:07.589655 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.590291 21281 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 03:00:07.584687 24466 log.cc:927] Time spent T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20251214 03:00:07.591591 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 268178us
I20251214 03:00:07.599740 21098 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.602546 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.606870 21173 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.589003 24458 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:07.595395 24478 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:07.497413 24415 consensus_queue.cc:237] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.615131 21173 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:07.617805 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:07.619652 24264 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:07.620472 22012 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:07.621708 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.622386 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.622846 21275 raft_consensus.cc:3055] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Stepping down as leader of term 8
I20251214 03:00:07.623174 21275 raft_consensus.cc:740] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.622942 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.623425 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.623808 21280 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:07.623003 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.624058 21275 consensus_queue.cc:260] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.625355 21275 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.621886 24493 consensus_queue.cc:237] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.635691 24394 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:07.615144 24478 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.635318 21131 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 03:00:07.639137 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.592202 24463 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.639714 21276 raft_consensus.cc:3060] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:07.639815 22014 raft_consensus.cc:1275] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251214 03:00:07.640450 21248 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.641288 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.642261 21208 raft_consensus.cc:2468] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 03:00:07.611304 24420 raft_consensus.cc:2749] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 03:00:07.644872 21201 raft_consensus.cc:1240] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20251214 03:00:07.642215 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:07.645080 21281 raft_consensus.cc:1240] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:07.646608 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:07.646987 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 165274us
I20251214 03:00:07.647794 24460 raft_consensus.cc:3055] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:07.615581 24491 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:07.648620 24491 raft_consensus.cc:697] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:07.649719 24400 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:07.649482 24491 consensus_queue.cc:237] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.650120 24400 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:07.650504 24400 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.599573 24492 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20251214 03:00:07.655843 21276 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:07.648171 24460 raft_consensus.cc:740] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.658851 21985 leader_election.cc:304] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:07.660604 21985 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.662487 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.654409 24489 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.665548 24394 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:07.665763 24472 consensus_queue.cc:1059] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:07.667408 24472 raft_consensus.cc:3055] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:07.667726 24472 raft_consensus.cc:740] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.616022 24477 consensus_queue.cc:237] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.667850 24413 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.668551 24472 consensus_queue.cc:260] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.669677 24472 raft_consensus.cc:3060] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.672900 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.637470 24478 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:07.705109 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 58609.9us
I20251214 03:00:07.673063 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.712865 21202 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:07.613057 24458 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.714233 24495 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
W20251214 03:00:07.719295 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.703545 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:07.720007 21096 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.722486 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.723197 21201 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:07.730705 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:07.731288 22018 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 8
I20251214 03:00:07.732209 21985 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.733119 21985 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.671880 24396 raft_consensus.cc:2749] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 03:00:07.671432 24421 raft_consensus.cc:2749] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:07.669800 24413 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:07.671463 24282 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:07.740999 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.746191 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.741775 21099 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:07.707775 24454 raft_consensus.cc:2804] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:07.761283 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.752010 24450 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:07.760164 21275 raft_consensus.cc:2410] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20251214 03:00:07.758044 21986 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:07.733927 24407 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.639401 24483 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.730165 24430 raft_consensus.cc:2749] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 03:00:07.733963 24481 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:07.764456 22018 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:07.744397 24448 raft_consensus.cc:2804] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:07.764241 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.767644 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:07.764938 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 155708us
I20251214 03:00:07.740507 24478 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:07.781441 21248 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:07.785151 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.785570 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.755968 24282 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.763482 24396 raft_consensus.cc:2804] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:07.763854 24462 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:07.763681 24454 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.763417 24455 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:07.764009 24450 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:07.791918 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:07.793357 21247 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.794207 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.766553 24458 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.795275 24454 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:07.793193 24448 raft_consensus.cc:697] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:07.793339 24462 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 03:00:07.792275 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 59235.8us
W20251214 03:00:07.796007 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:07.796139 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.782456 24478 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:07.782213 24463 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.788385 24394 raft_consensus.cc:2749] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:07.828528 21124 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:07.795166 24450 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:07.788662 24282 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:07.782953 24468 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:07.786623 24415 raft_consensus.cc:2804] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader election won for term 5
I20251214 03:00:07.792613 24364 raft_consensus.cc:2749] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:07.798203 21028 catalog_manager.cc:5654] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 5 to 6, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 6 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 03:00:07.843677 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59016 (request call id 175) took 1102 ms. Trace:
I20251214 03:00:07.844177 21028 catalog_manager.cc:5654] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 8 to 9, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 9 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 03:00:07.846447 21028 catalog_manager.cc:5654] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 721d604130924d939d291fe3eef8f6d1 (127.19.131.65). New cstate: current_term: 7 leader_uuid: "721d604130924d939d291fe3eef8f6d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 03:00:07.813073 21245 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:07.794857 24493 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:07.788784 24485 raft_consensus.cc:2804] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:07.811789 24462 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
W20251214 03:00:07.806763 24436 consensus_meta.cc:287] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.765s user 0.009s sys 0.001s
I20251214 03:00:07.811386 24448 consensus_queue.cc:237] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.830664 24264 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 03:00:07.869525 24485 raft_consensus.cc:697] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [term 7 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.831988 24471 raft_consensus.cc:993] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 03:00:07.832068 24483 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.833102 24511 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:07.760877 24460 consensus_queue.cc:260] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.771878 24407 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.796916 24396 raft_consensus.cc:697] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:07.825929 24478 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:07.808432 24495 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:07.802093 24482 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:07.849368 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:07.804582 24508 log.cc:927] Time spent T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20251214 03:00:07.852625 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.854197 24463 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20251214 03:00:07.844101 21027 rpcz_store.cc:276] 1214 03:00:06.741094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:06.741201 (+ 107us) service_pool.cc:225] Handling call
1214 03:00:07.843612 (+1102411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 03:00:07.218705 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 03:00:07.315013 (+ 96308us) write_op.cc:432] Acquiring schema lock in shared mode
1214 03:00:07.315047 (+ 34us) write_op.cc:435] Acquired schema lock
1214 03:00:07.315063 (+ 16us) tablet.cc:660] Decoding operations
1214 03:00:07.315373 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
1214 03:00:07.315431 (+ 58us) write_op.cc:641] Partition lock acquired for write op
1214 03:00:07.315458 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
1214 03:00:07.315625 (+ 167us) tablet.cc:699] Row locks acquired
1214 03:00:07.315639 (+ 14us) write_op.cc:260] PREPARE: finished
1214 03:00:07.315774 (+ 135us) write_op.cc:270] Start()
1214 03:00:07.315875 (+ 101us) write_op.cc:276] Timestamp: P: 1765681207315728 usec, L: 0
1214 03:00:07.315893 (+ 18us) op_driver.cc:348] REPLICATION: starting
1214 03:00:07.316523 (+ 630us) log.cc:844] Serialized 536 byte log entry
1214 03:00:07.627582 (+311059us) op_driver.cc:464] REPLICATION: finished
1214 03:00:07.810151 (+182569us) write_op.cc:301] APPLY: starting
1214 03:00:07.810231 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
1214 03:00:07.810323 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
1214 03:00:07.810451 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 03:00:07.810738 (+ 287us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 03:00:07.810914 (+ 176us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 03:00:07.810959 (+ 45us) write_op.cc:312] APPLY: finished
1214 03:00:07.811138 (+ 179us) log.cc:844] Serialized 23 byte log entry
1214 03:00:07.839410 (+ 28272us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:07.839504 (+ 94us) write_op.cc:454] Released schema lock
1214 03:00:07.839547 (+ 43us) write_op.cc:341] FINISH: Updating metrics
1214 03:00:07.839994 (+ 447us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:07.840028 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":178372,"apply.run_cpu_time_us":1808,"apply.run_wall_time_us":29814,"num_ops":1,"prepare.queue_time_us":338456,"prepare.run_cpu_time_us":27270,"prepare.run_wall_time_us":281883,"raft.queue_time_us":130832,"raft.run_cpu_time_us":17111,"raft.run_wall_time_us":147393,"replication_time_us":311531,"thread_start_us":571648,"threads_started":4,"wal-append.queue_time_us":153909}]]}
I20251214 03:00:07.856796 24458 consensus_queue.cc:1059] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:07.859259 24467 consensus_queue.cc:1059] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:07.859896 21247 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.804718 24400 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.902817 24464 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20251214 03:00:07.903617 24464 consensus_queue.cc:1243] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 721d604130924d939d291fe3eef8f6d1 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251214 03:00:07.904152 24400 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 03:00:07.875936 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 57430.5us
I20251214 03:00:07.907364 21985 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.907647 24450 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.908581 24495 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:07.908982 24495 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:07.909322 24495 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:07.909670 24450 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:07.868618 24413 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election won for term 8
I20251214 03:00:07.911589 24396 consensus_queue.cc:237] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.913153 24458 raft_consensus.cc:3055] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Stepping down as leader of term 9
I20251214 03:00:07.913548 24458 raft_consensus.cc:740] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.913677 24467 raft_consensus.cc:3055] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Stepping down as leader of term 8
I20251214 03:00:07.914065 24467 raft_consensus.cc:740] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.877394 24364 consensus_queue.cc:1048] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20251214 03:00:07.914367 24458 consensus_queue.cc:260] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.914829 24467 consensus_queue.cc:260] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.879698 24511 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:07.916038 24511 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:07.916663 24479 raft_consensus.cc:2749] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:07.917079 24407 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.913185 24463 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Stepping down as leader of term 10
I20251214 03:00:07.917590 24463 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.877163 24515 consensus_queue.cc:1048] T 19fd2760f37a43e2983b015db150e285 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20251214 03:00:07.918335 24463 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:07.889609 21202 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.919464 24463 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:07.881201 24380 raft_consensus.cc:1081] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Signalling peer 6e36caefa37d47a5968cffd9e270044f to start an election
I20251214 03:00:07.895272 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.895458 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.896432 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.895905 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.896282 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.871300 24436 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.918686 24413 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:07.921969 24485 consensus_queue.cc:237] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.883863 22018 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:07.926491 21202 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
W20251214 03:00:07.926837 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 67021.4us
I20251214 03:00:07.927987 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:07.869684 24493 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.932332 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:07.932471 21248 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:07.932781 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.924611 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.934698 21206 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.937978 21277 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5751534b61ed4568abd62057238d5fda"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:43308
I20251214 03:00:07.938516 21277 raft_consensus.cc:606] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:07.938887 21277 raft_consensus.cc:614] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251214 03:00:07.941437 24436 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:07.942391 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:07.942266 24413 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.885061 24415 raft_consensus.cc:697] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:07.945600 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:07.947422 19981 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251214 03:00:07.948230 24493 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:07.949215 19981 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251214 03:00:07.948899 24454 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.950244 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 03:00:07.904150 24464 raft_consensus.cc:3055] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Stepping down as leader of term 4
I20251214 03:00:07.957166 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.959507 24407 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:07.956700 21202 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 10.
I20251214 03:00:07.960705 21279 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:07.962442 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:07.965430 24415 consensus_queue.cc:237] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:07.960695 21273 raft_consensus.cc:2376] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6. Re-sending same reply.
I20251214 03:00:07.944232 24468 raft_consensus.cc:2749] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251214 03:00:07.985702 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:07.986457 21126 raft_consensus.cc:2393] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 10: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.975427 21206 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:07.982216 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 03:00:07.988818 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 103303us
I20251214 03:00:07.946410 24515 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:07.989679 21248 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:07.954808 24458 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:07.982519 24476 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:07.946262 24467 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:07.992077 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:07.996151 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.988963 22014 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:07.995414 21208 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 10. Current term is 11. Ops: []
I20251214 03:00:07.994184 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.997391 21126 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:07.998175 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.998777 21133 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 03:00:08.000038 24472 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:07.990613 24377 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:08.002425 24514 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20251214 03:00:08.002157 24377 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.004951 24377 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.005707 24282 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.017876 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:07.987658 24436 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 03:00:08.006758 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.018653 21273 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:07.960513 24464 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:08.016528 21124 raft_consensus.cc:2393] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:08.021674 22014 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251214 03:00:08.018785 21278 raft_consensus.cc:2410] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20251214 03:00:07.948866 24465 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20251214 03:00:08.019124 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.019791 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.017755 21096 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:08.022272 21985 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:08.023058 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.023845 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:07.987968 24407 raft_consensus.cc:697] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:08.022174 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.050113 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.027519 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.051137 21277 raft_consensus.cc:2393] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 6: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:07.991305 24492 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:08.028394 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.029050 21126 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:08.027366 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.053011 21199 raft_consensus.cc:1240] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:08.029898 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.053818 21133 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:08.029927 21125 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:08.029317 21279 raft_consensus.cc:2376] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6. Re-sending same reply.
I20251214 03:00:08.055050 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.029699 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.056996 21205 raft_consensus.cc:2393] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:08.035972 21278 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92"
dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
from {username='slave'} at 127.0.0.1:50530
I20251214 03:00:08.047327 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:08.043010 21247 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.048161 22018 raft_consensus.cc:1240] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:08.050390 21130 raft_consensus.cc:1240] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
I20251214 03:00:08.061209 24361 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:08.061609 24282 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.062740 24429 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:08.063149 24429 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.063463 24429 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:08.047649 24436 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:08.066843 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.067375 24436 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.021167 24477 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:08.075043 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43967us
I20251214 03:00:08.057461 21278 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
W20251214 03:00:08.075714 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 47179.4us
I20251214 03:00:08.017395 24472 raft_consensus.cc:697] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [term 9 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:08.070091 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.073941 21126 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:08.079334 21278 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:08.080358 21124 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:08.082413 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.084278 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.056455 24397 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:08.093771 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.070072 24400 raft_consensus.cc:2749] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:08.014029 24476 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.101146 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:08.102273 21281 raft_consensus.cc:2468] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:08.107769 21205 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 03:00:08.108187 21098 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:08.076566 24361 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:08.092902 24450 raft_consensus.cc:2749] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 03:00:08.120764 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.069631 24407 consensus_queue.cc:237] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:08.205642 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:08.126341 21278 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.110289 24397 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:08.114183 24377 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
W20251214 03:00:08.227200 21246 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:08.112671 24470 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20251214 03:00:08.080039 24477 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.243803 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:08.247534 21199 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:08.070763 24492 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.232931 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.237906 21278 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:08.238530 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.120599 24454 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:08.223814 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 114334us
I20251214 03:00:08.246253 22014 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251214 03:00:08.299373 24492 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:08.332250 21126 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251214 03:00:08.281129 24397 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.312423 24502 raft_consensus.cc:2749] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:08.315698 21132 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 03:00:08.121376 24361 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.320742 21208 raft_consensus.cc:1275] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251214 03:00:08.299340 21098 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:08.313902 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.372187 21205 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:08.202741 24478 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.311056 24477 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:08.296342 24476 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.234786 24413 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:08.297895 24523 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20251214 03:00:08.340480 21245 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:08.357633 21133 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:08.304261 24462 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.242774 24472 consensus_queue.cc:237] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.300133 24465 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:08.331369 24429 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.405757 24429 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:08.247782 24377 raft_consensus.cc:697] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:08.369982 21124 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:08.372478 21985 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.374282 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 03:00:08.407538 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 204318us
W20251214 03:00:08.375312 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 76049.8us
I20251214 03:00:08.377177 21199 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:08.410846 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:08.411072 21098 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:08.380373 21280 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 03:00:08.379561 24413 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.313855 24495 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.377916 24370 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 03:00:08.406891 22013 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 03:00:08.409483 21282 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 11.
I20251214 03:00:08.411312 22014 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:08.396349 24511 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.413156 24377 consensus_queue.cc:237] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.415694 24478 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:08.418576 21274 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:08.422607 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:08.433300 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:08.433967 22012 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:08.399870 24361 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:08.436733 24478 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:08.437798 24460 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:08.438331 24510 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:08.438907 24510 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.426923 24454 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20251214 03:00:08.440944 24510 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.421164 24429 raft_consensus.cc:2749] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:08.452728 21028 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51636 (request call id 282) took 1484 ms. Trace:
I20251214 03:00:08.422605 24516 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20251214 03:00:08.388980 24477 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20251214 03:00:08.475253 22014 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:08.441900 24513 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251214 03:00:08.433018 24482 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20251214 03:00:08.478210 21096 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 03:00:08.478965 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 46880.6us
I20251214 03:00:08.481060 21208 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:08.482924 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.481189 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.483394 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:08.483839 21199 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:08.483516 21203 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 12.
I20251214 03:00:08.408607 24465 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.453164 21028 rpcz_store.cc:276] 1214 03:00:06.968325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:06.968463 (+ 138us) service_pool.cc:225] Handling call
1214 03:00:08.452653 (+1484190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 03:00:07.886450 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 03:00:07.907053 (+ 20603us) write_op.cc:432] Acquiring schema lock in shared mode
1214 03:00:07.907093 (+ 40us) write_op.cc:435] Acquired schema lock
1214 03:00:07.907106 (+ 13us) tablet.cc:660] Decoding operations
1214 03:00:07.944691 (+ 37585us) write_op.cc:620] Acquiring the partition lock for write op
1214 03:00:07.950073 (+ 5382us) write_op.cc:641] Partition lock acquired for write op
1214 03:00:07.954498 (+ 4425us) tablet.cc:683] Acquiring locks for 3 operations
1214 03:00:07.959213 (+ 4715us) tablet.cc:699] Row locks acquired
1214 03:00:07.959232 (+ 19us) write_op.cc:260] PREPARE: finished
1214 03:00:07.959373 (+ 141us) write_op.cc:270] Start()
1214 03:00:07.959477 (+ 104us) write_op.cc:276] Timestamp: P: 1765681207959329 usec, L: 0
1214 03:00:07.959499 (+ 22us) op_driver.cc:348] REPLICATION: starting
1214 03:00:07.960059 (+ 560us) log.cc:844] Serialized 1302 byte log entry
1214 03:00:08.017203 (+ 57144us) op_driver.cc:464] REPLICATION: finished
1214 03:00:08.052863 (+ 35660us) write_op.cc:301] APPLY: starting
1214 03:00:08.061265 (+ 8402us) tablet.cc:1365] starting BulkCheckPresence
1214 03:00:08.061391 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
1214 03:00:08.061488 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 03:00:08.061926 (+ 438us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 03:00:08.062086 (+ 160us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1214 03:00:08.062125 (+ 39us) write_op.cc:312] APPLY: finished
1214 03:00:08.062305 (+ 180us) log.cc:844] Serialized 35 byte log entry
1214 03:00:08.062456 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:08.062543 (+ 87us) write_op.cc:454] Released schema lock
1214 03:00:08.062612 (+ 69us) write_op.cc:341] FINISH: Updating metrics
1214 03:00:08.062942 (+ 330us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:08.062968 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":7713,"apply.run_cpu_time_us":1775,"apply.run_wall_time_us":10049,"num_ops":3,"prepare.queue_time_us":2393,"prepare.run_cpu_time_us":3494,"prepare.run_wall_time_us":79400,"replication_time_us":28347}]]}
I20251214 03:00:08.485909 21098 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:08.489039 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.489629 21205 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 03:00:08.491971 21247 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.493088 21247 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.493541 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:08.493952 24463 raft_consensus.cc:2749] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:08.494633 24463 raft_consensus.cc:2749] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:08.498116 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.498416 21247 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.499500 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:08.500036 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.500560 21278 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 12.
I20251214 03:00:08.500502 21247 leader_election.cc:304] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.502485 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.507252 21247 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:08.520306 21201 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 03:00:08.521533 21205 raft_consensus.cc:1275] T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251214 03:00:08.524784 24444 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:08.525565 24444 consensus_queue.cc:1243] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer 721d604130924d939d291fe3eef8f6d1 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251214 03:00:08.486860 24380 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 03:00:08.529277 22018 raft_consensus.cc:1217] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[4.6-4.6] Dedup: 4.6->[]
I20251214 03:00:08.529727 21203 raft_consensus.cc:1275] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251214 03:00:08.529932 22018 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251214 03:00:08.529755 21248 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:08.491819 24370 raft_consensus.cc:2804] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:08.531260 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.531478 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.536127 21280 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:08.504667 24471 raft_consensus.cc:2804] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:08.513260 24407 raft_consensus.cc:2749] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 03:00:08.504885 24463 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:08.504525 24459 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:08.544518 21203 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 03:00:08.527019 24400 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20251214 03:00:08.489626 24478 consensus_queue.cc:1059] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20251214 03:00:08.552052 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 58949.4us
I20251214 03:00:08.553169 21248 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:08.532262 24467 raft_consensus.cc:2749] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:08.552592 24477 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:08.547530 24370 raft_consensus.cc:697] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:08.535758 24485 consensus_queue.cc:1048] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20251214 03:00:08.599965 22018 raft_consensus.cc:1240] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:08.546687 24502 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.045s
I20251214 03:00:08.565310 24462 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.600152 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.549993 24396 consensus_queue.cc:1048] T 4b611d0c068c451bb4eeb7eda4823036 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
W20251214 03:00:08.600916 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 109151us
I20251214 03:00:08.565358 24471 raft_consensus.cc:697] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:08.601755 24477 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.569558 24407 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:08.547395 24437 consensus_queue.cc:1048] T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20251214 03:00:08.594336 24489 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:08.595914 24413 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:08.594513 24400 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20251214 03:00:08.607720 21203 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 12.
I20251214 03:00:08.616604 24511 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.617369 24495 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:08.636516 24489 raft_consensus.cc:697] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Becoming Leader. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:08.637408 24489 consensus_queue.cc:237] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.639422 21205 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 03:00:08.641142 24464 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.642369 24464 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:08.643519 21279 raft_consensus.cc:2468] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:08.630657 24407 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.645095 24471 consensus_queue.cc:237] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.646991 24407 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:08.635246 24538 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:08.621974 24477 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:08.611641 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.633160 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:08.656045 21282 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 03:00:08.658877 21274 raft_consensus.cc:1275] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 03:00:08.659978 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:08.630425 24502 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:08.636165 24526 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20251214 03:00:08.635967 24548 consensus_queue.cc:1059] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20251214 03:00:08.627417 22012 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:08.567797 24436 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20251214 03:00:08.661758 21173 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:08.661957 21282 raft_consensus.cc:1275] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251214 03:00:08.633157 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:08.667074 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:08.664036 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:08.659930 24407 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20251214 03:00:08.669203 24502 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:08.763906 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 130989us
I20251214 03:00:08.787575 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:08.764540 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:08.656971 24377 consensus_queue.cc:1059] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20251214 03:00:08.655668 24465 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:08.799710 21124 raft_consensus.cc:1240] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:08.803149 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:08.806272 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 177343us
I20251214 03:00:08.763602 24407 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.765081 24488 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:08.763586 24548 raft_consensus.cc:3055] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:08.815913 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:08.820302 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:08.820874 21207 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:08.646929 24370 consensus_queue.cc:237] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.808601 24380 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20251214 03:00:08.811476 24460 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:08.812774 24407 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Advancing to term 13
I20251214 03:00:08.660935 24538 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:08.849050 21207 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:08.813663 24548 raft_consensus.cc:740] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 03:00:08.849962 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 85517.4us
I20251214 03:00:08.852505 24456 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.187s
I20251214 03:00:08.853827 24534 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:08.854288 24534 raft_consensus.cc:493] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.854631 24534 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:08.858068 24537 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:08.858484 24537 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.858836 24537 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:08.812037 24455 consensus_queue.cc:1059] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20251214 03:00:08.840993 24380 raft_consensus.cc:493] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:08.841558 24429 raft_consensus.cc:3055] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
W20251214 03:00:08.907226 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 56071.3us
W20251214 03:00:08.852003 24458 consensus_meta.cc:287] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.855s user 0.007s sys 0.002s
I20251214 03:00:08.847630 24465 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:08.859133 24548 consensus_queue.cc:260] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:08.999713 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 146069us
I20251214 03:00:08.917536 24429 raft_consensus.cc:740] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:08.937211 24380 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Advancing to term 13
I20251214 03:00:08.850121 24460 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.035152 21275 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 10.
I20251214 03:00:08.990895 24444 raft_consensus.cc:493] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:08.944512 24456 consensus_queue.cc:1048] T 75c047f1d6d94237afc050db2241d659 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20251214 03:00:09.017422 24458 raft_consensus.cc:493] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:09.021090 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 53557.8us
I20251214 03:00:09.117681 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:08.827534 24488 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:08.993603 24465 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.072304 24548 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:09.148490 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 198494us
I20251214 03:00:09.078048 24429 consensus_queue.cc:260] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.158350 24429 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
W20251214 03:00:09.174363 21098 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.013s sys 0.015s
W20251214 03:00:09.174813 21098 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.013s sys 0.015s
I20251214 03:00:09.175626 21096 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 03:00:09.179512 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 185250us
I20251214 03:00:09.084532 24549 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20251214 03:00:09.167500 24552 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.360s user 0.000s sys 0.001s
I20251214 03:00:09.200379 21985 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:09.083436 24444 raft_consensus.cc:515] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.117681 24458 raft_consensus.cc:515] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:09.292158 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 68000.8us
I20251214 03:00:09.210714 24545 consensus_queue.cc:1048] T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.107s
I20251214 03:00:09.318189 21099 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:09.233568 21280 raft_consensus.cc:1275] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20251214 03:00:09.231189 24553 log.cc:927] Time spent T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
W20251214 03:00:09.343214 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 142943us
I20251214 03:00:09.212181 24549 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 03:00:09.339205 24565 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
W20251214 03:00:09.423591 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 105492us
I20251214 03:00:09.340401 24377 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:09.225247 24502 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:09.460604 21279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 3184) took 1410 ms (client timeout 1539 ms). Trace:
I20251214 03:00:09.419924 24549 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Advancing to term 11
W20251214 03:00:09.461061 21279 rpcz_store.cc:269] 1214 03:00:08.037388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.054823 (+ 17435us) service_pool.cc:225] Handling call
1214 03:00:09.447214 (+1392391us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400331b54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 03:00:09.447331 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1352014592}
I20251214 03:00:09.424139 24394 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.486940 19981 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251214 03:00:09.487795 21202 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20251214 03:00:09.445210 24421 raft_consensus.cc:2749] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251214 03:00:09.446731 24502 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:09.452664 24377 raft_consensus.cc:697] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:09.451371 24538 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:09.462898 24462 raft_consensus.cc:493] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:09.522589 24555 log.cc:927] Time spent T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.443s user 0.001s sys 0.000s
I20251214 03:00:09.522871 24572 raft_consensus.cc:2804] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 03:00:09.524576 24566 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:09.525130 24566 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.522622 24415 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.527061 24566 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:09.527145 24415 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:09.527628 24551 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.611s user 0.000s sys 0.001s
W20251214 03:00:09.516228 24464 consensus_meta.cc:287] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.873s user 0.005s sys 0.004s
I20251214 03:00:09.502872 24460 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:09.539570 24488 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:09.540026 24559 log.cc:927] Time spent T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.491s user 0.001s sys 0.000s
I20251214 03:00:09.540468 24562 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:09.541885 24563 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.105s user 0.001s sys 0.001s
I20251214 03:00:09.541712 24380 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.517247 24502 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.546782 24561 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:09.547089 24556 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.464s user 0.000s sys 0.001s
I20251214 03:00:09.547231 24458 leader_election.cc:290] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:09.542737 24377 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:09.548640 24407 consensus_meta.cc:287] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.688s user 0.007s sys 0.002s
I20251214 03:00:09.529843 24462 raft_consensus.cc:515] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.551720 24465 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:09.555264 24415 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:09.596311 24459 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:09.560245 24474 raft_consensus.cc:3055] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:09.499534 24394 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.567930 21204 raft_consensus.cc:1275] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20251214 03:00:09.550230 24534 consensus_meta.cc:287] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.695s user 0.008s sys 0.000s
W20251214 03:00:09.560178 24477 consensus_meta.cc:287] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f: Time spent flushing consensus metadata: real 0.894s user 0.008s sys 0.000s
I20251214 03:00:09.573982 21199 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 03:00:09.565516 24572 raft_consensus.cc:697] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
W20251214 03:00:09.583330 24537 consensus_meta.cc:287] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.724s user 0.006s sys 0.002s
I20251214 03:00:09.550033 24576 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:09.791296 24576 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 03:00:09.567222 24538 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.534623 24421 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20251214 03:00:09.793467 24538 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.570276 24564 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.794852 24564 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.574476 24364 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.797703 24364 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.799572 24364 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:09.592046 24380 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:09.803817 24572 consensus_queue.cc:237] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.597698 24462 leader_election.cc:290] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:09.733916 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 279232us
I20251214 03:00:09.734194 21204 pending_rounds.cc:85] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Aborting all ops after (but not including) 5
I20251214 03:00:09.787740 24477 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:09.790711 24558 log.cc:927] Time spent T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.534s user 0.004s sys 0.000s
I20251214 03:00:09.590296 24407 raft_consensus.cc:515] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.811280 21204 pending_rounds.cc:107] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20251214 03:00:09.916057 21204 raft_consensus.cc:2889] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 03:00:09.917045 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 3298) took 1362 ms. Trace:
I20251214 03:00:09.733686 24474 raft_consensus.cc:740] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:09.937151 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1708) took 1253 ms. Trace:
I20251214 03:00:09.968537 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 597) took 1316 ms. Trace:
W20251214 03:00:09.847934 24429 consensus_meta.cc:287] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.686s user 0.005s sys 0.004s
I20251214 03:00:09.917475 21206 rpcz_store.cc:276] 1214 03:00:08.481259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.483706 (+ 2447us) service_pool.cc:225] Handling call
1214 03:00:08.493502 (+ 9796us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.506519 (+ 13017us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.506538 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.527030 (+ 20492us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.613977 (+ 86947us) raft_consensus.cc:1645] Marking committed up to 5
1214 03:00:08.614014 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.614088 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:09.843683 (+1229595us) raft_consensus.cc:1685] finished
1214 03:00:09.843707 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:09.843860 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16528,"raft.queue_time_us":13145,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":18792,"thread_start_us":59098,"threads_started":1,"wal-append.queue_time_us":270840}
I20251214 03:00:09.998054 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 598) took 1276 ms. Trace:
I20251214 03:00:10.007033 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:10.007725 21199 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:09.998517 21208 rpcz_store.cc:276] 1214 03:00:08.721641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.721807 (+ 166us) service_pool.cc:225] Handling call
1214 03:00:08.722174 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:09.071685 (+349511us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:09.071713 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:09.078161 (+ 6448us) log.cc:844] Serialized 26 byte log entry
1214 03:00:09.422685 (+344524us) raft_consensus.cc:1645] Marking committed up to 7
1214 03:00:09.422724 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:09.968154 (+545430us) spinlock_profiling.cc:243] Waited 85.4 ms on lock 0x7b5400021f54. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 03:00:09.968198 (+ 44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:09.968250 (+ 52us) raft_consensus.cc:1685] finished
1214 03:00:09.968265 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:09.997981 (+ 29716us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16398,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4468,"spinlock_wait_cycles":196384640,"thread_start_us":525103,"threads_started":2,"wal-append.queue_time_us":420106}
I20251214 03:00:10.013687 21273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50530 (request call id 3179) took 1988 ms. Trace:
I20251214 03:00:09.575161 24562 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.968993 21205 rpcz_store.cc:276] 1214 03:00:08.652040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.652237 (+ 197us) service_pool.cc:225] Handling call
1214 03:00:08.662261 (+ 10024us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.662682 (+ 421us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.662713 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.667390 (+ 4677us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.824824 (+157434us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:08.824866 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.824933 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:09.968353 (+1143420us) raft_consensus.cc:1685] finished
1214 03:00:09.968371 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:09.968473 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":151,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":129275,"thread_start_us":61517,"threads_started":1,"wal-append.queue_time_us":176222}
I20251214 03:00:10.017736 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:09.581035 24561 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.022716 21208 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 03:00:09.986079 24566 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.760274 24459 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:10.031350 21098 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.006s sys 0.005s
W20251214 03:00:10.031672 21282 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.000s sys 0.012s
W20251214 03:00:10.031895 21246 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.001s sys 0.010s
W20251214 03:00:09.929442 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:09.768740 24534 raft_consensus.cc:515] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:09.937693 21201 rpcz_store.cc:276] 1214 03:00:08.684144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.685777 (+ 1633us) service_pool.cc:225] Handling call
1214 03:00:08.801321 (+115544us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.803538 (+ 2217us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.803567 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.807250 (+ 3683us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.986777 (+179527us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:08.986816 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.986883 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:09.932202 (+945319us) raft_consensus.cc:1685] finished
1214 03:00:09.932228 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:09.937086 (+ 4858us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13161,"raft.queue_time_us":2318,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4333,"thread_start_us":162656,"threads_started":1,"wal-append.queue_time_us":406185}
I20251214 03:00:10.006678 24282 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:09.845675 24537 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:09.968257 24564 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.464834 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:10.466652 21208 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 03:00:10.345024 24502 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:10.196396 21171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.328s user 0.010s sys 0.022s
I20251214 03:00:10.289987 24585 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.014113 21273 rpcz_store.cc:276] 1214 03:00:08.004496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.019329 (+ 14833us) service_pool.cc:225] Handling call
1214 03:00:08.055684 (+ 36355us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.056051 (+ 367us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.056079 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.457243 (+401164us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b2c00107c08. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670d89c9b1 00007f670d8abfe2 00007f670efd9851 00007f670ef9e0ba 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22
1214 03:00:08.457684 (+ 441us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.534136 (+ 76452us) raft_consensus.cc:1645] Marking committed up to 7
1214 03:00:08.534169 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.534229 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:08.616276 (+ 82047us) raft_consensus.cc:1685] finished
1214 03:00:08.616296 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:09.993079 (+1376783us) spinlock_profiling.cc:243] Waited 5.66 ms on lock 0x7b54000483d0. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9d52d 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 03:00:09.993230 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1231,"raft.queue_time_us":2016,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6625,"spinlock_wait_cycles":53600000,"thread_start_us":76118,"threads_started":1,"wal-append.queue_time_us":146448,"wal-append.run_cpu_time_us":1387,"wal-append.run_wall_time_us":1009046}
I20251214 03:00:10.018321 24576 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.338307 24450 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.284487 24587 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 03:00:10.266489 24400 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.015s sys 0.024s
I20251214 03:00:10.471942 21199 raft_consensus.cc:3060] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:10.218366 21247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.019s sys 0.040s
W20251214 03:00:10.472970 24400 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.872s user 0.018s sys 0.026s
W20251214 03:00:10.470898 21171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.607s user 0.013s sys 0.029s
W20251214 03:00:10.428398 24549 consensus_meta.cc:287] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.931s user 0.008s sys 0.000s
I20251214 03:00:10.289618 24562 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.474618 24562 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:10.475512 21246 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.002s sys 0.015s
W20251214 03:00:10.383221 21985 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.007s sys 0.011s
W20251214 03:00:10.480724 21985 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.136s user 0.007s sys 0.011s
W20251214 03:00:10.481033 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 1.13633e+06us
I20251214 03:00:10.027580 22013 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44272 (request call id 521) took 1417 ms. Trace:
I20251214 03:00:10.482772 22013 rpcz_store.cc:276] 1214 03:00:08.610021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.610406 (+ 385us) service_pool.cc:225] Handling call
1214 03:00:08.615222 (+ 4816us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.619177 (+ 3955us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.619205 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.619630 (+ 425us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.825118 (+205488us) raft_consensus.cc:1645] Marking committed up to 8
1214 03:00:08.825155 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.825216 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:09.558099 (+732883us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 03:00:10.027306 (+469207us) raft_consensus.cc:1685] finished
1214 03:00:10.027330 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:10.027517 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3571,"raft.queue_time_us":4253,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"spinlock_wait_cycles":7808896,"thread_start_us":205159,"threads_started":1,"wal-append.queue_time_us":249800}
W20251214 03:00:10.486678 21282 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.276s user 0.001s sys 0.017s
W20251214 03:00:10.107172 21207 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.006s sys 0.009s
W20251214 03:00:10.473232 21247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.991s user 0.019s sys 0.049s
W20251214 03:00:10.058967 21986 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.028s sys 0.043s
I20251214 03:00:10.491276 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:10.474295 24282 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:10.059116 21983 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.029s sys 0.041s
W20251214 03:00:10.569157 21983 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s user 0.051s sys 0.059s
I20251214 03:00:10.569311 21206 raft_consensus.cc:3060] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Advancing to term 13
I20251214 03:00:10.525547 21199 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20251214 03:00:10.568562 21986 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s user 0.040s sys 0.070s
W20251214 03:00:10.514374 21207 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.581s user 0.007s sys 0.013s
W20251214 03:00:10.472106 21098 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.287s user 0.007s sys 0.010s
W20251214 03:00:10.438525 21097 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.001s sys 0.011s
W20251214 03:00:10.573437 21097 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.389s user 0.006s sys 0.015s
W20251214 03:00:10.571158 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 1.04386e+06us
W20251214 03:00:10.578926 21245 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.036s sys 0.019s
W20251214 03:00:10.579299 21245 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.036s sys 0.019s
W20251214 03:00:10.218297 24377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.015s sys 0.030s
I20251214 03:00:10.582525 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:10.583182 21199 raft_consensus.cc:3060] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:10.471303 24534 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:10.266249 24436 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.010s sys 0.022s
I20251214 03:00:10.588282 21200 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 6. Current term is 7. Ops: []
W20251214 03:00:10.046046 24370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.011s
I20251214 03:00:10.194717 24578 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.472936 24587 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.419878 24458 raft_consensus.cc:493] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.076062 24477 leader_election.cc:290] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.196839 24397 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.285732 24584 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.288424 24561 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.344921 24364 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.287312 24394 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.176532 24581 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.470729 24502 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:10.568974 21248 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.011s sys 0.008s
W20251214 03:00:10.608376 21248 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s user 0.011s sys 0.008s
I20251214 03:00:10.471426 24564 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.491276 24571 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.181200 24481 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.611891 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.612490 22011 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 03:00:10.108160 24464 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.438125 24573 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.576825 24576 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:10.587999 24377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.994s user 0.023s sys 0.051s
I20251214 03:00:10.076668 24538 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:10.593693 24370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.070s user 0.000s sys 0.018s
W20251214 03:00:10.593459 24436 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.029s user 0.011s sys 0.024s
I20251214 03:00:10.472419 24450 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.469571 24566 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.474025 24549 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.618973 24564 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.619830 24477 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.471742 24585 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.641450 24586 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:10.645884 24586 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.606858 24584 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.647817 24586 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.648485 24584 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.648983 24584 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.631044 24464 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.596753 24413 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.475107 24562 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.605299 24595 raft_consensus.cc:493] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.614118 24407 leader_election.cc:290] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.619632 24593 raft_consensus.cc:493] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.597764 24578 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:10.639554 24579 log.cc:927] Time spent T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.444s user 0.001s sys 0.001s
I20251214 03:00:10.604182 24458 raft_consensus.cc:515] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.078673 24474 consensus_queue.cc:260] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.644646 24549 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.648870 24586 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:10.640856 24538 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.644455 24566 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.644533 24564 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.621384 24502 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.656229 24585 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.619293 24581 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.649444 24584 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.636941 24576 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.627288 24571 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.644799 24477 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.654819 24413 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.631937 24481 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.665938 24407 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.668995 24601 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.669682 24474 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:10.690259 24477 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.690460 24458 leader_election.cc:290] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.691933 24450 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.692643 24614 consensus_queue.cc:1048] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:10.692957 24502 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.684773 24608 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.693749 24608 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.694517 24413 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.695673 24608 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.696002 24576 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.697939 24459 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.698308 24584 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.668259 24586 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.699505 24578 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.700315 24586 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.702656 24581 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.703208 24407 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.675303 24585 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.705309 24464 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.705691 24585 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.706092 24587 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.706816 24537 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.658946 24595 raft_consensus.cc:515] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.691967 24481 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.707624 24585 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.705224 24407 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.709307 24595 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.660848 24593 raft_consensus.cc:515] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.714743 24593 leader_election.cc:290] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.667189 24538 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.716894 24538 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.717756 24601 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.719941 24601 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.727068 24562 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.633183 24573 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.732640 24573 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.598721 21199 raft_consensus.cc:2468] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:10.610810 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.610181 22014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.854734 24481 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.855772 24481 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.857326 24502 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:10.634230 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.858322 24502 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.859616 22018 raft_consensus.cc:2368] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 03:00:10.832548 21281 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 582) took 1487 ms. Trace:
I20251214 03:00:10.861651 24282 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.861286 21281 rpcz_store.cc:276] 1214 03:00:09.344373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:09.344492 (+ 119us) service_pool.cc:225] Handling call
1214 03:00:09.426917 (+ 82425us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:09.569737 (+142820us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:09.569771 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:09.581355 (+ 11584us) log.cc:844] Serialized 26 byte log entry
1214 03:00:10.468170 (+886815us) raft_consensus.cc:1645] Marking committed up to 8
1214 03:00:10.468213 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:10.468288 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:10.831759 (+363471us) raft_consensus.cc:1685] finished
1214 03:00:10.831804 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:10.832161 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":152671,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9015,"thread_start_us":593858,"threads_started":2,"wal-append.queue_time_us":672599}
I20251214 03:00:10.865937 21171 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:10.866550 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.867982 22013 raft_consensus.cc:2410] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20251214 03:00:10.868304 24573 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.865959 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.869138 24573 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:10.871063 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 296000us
I20251214 03:00:10.871331 24578 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.871469 24537 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 03:00:10.871933 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 293085us
I20251214 03:00:10.872188 24578 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.872417 24537 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.875269 24590 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:10.841732 24361 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.861459 24502 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.876019 24590 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.607918 24397 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.895946 24401 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:10.612051 24364 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.896922 24401 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.898308 21202 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 3310) took 1327 ms. Trace:
I20251214 03:00:10.899250 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:10.898767 21202 rpcz_store.cc:276] 1214 03:00:09.571088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:09.571380 (+ 292us) service_pool.cc:225] Handling call
1214 03:00:09.571674 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:09.796556 (+224882us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:09.796588 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:09.796629 (+ 41us) raft_consensus.cc:1645] Marking committed up to 5
1214 03:00:09.796659 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:10.897772 (+1101113us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b5400022454. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 03:00:10.897834 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:10.898084 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516718336}
I20251214 03:00:10.900942 24361 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.903199 24617 raft_consensus.cc:493] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 03:00:10.903452 21275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50530 (request call id 3170) took 3007 ms (client timeout 1739 ms). Trace:
I20251214 03:00:10.903826 24361 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.910127 22012 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 03:00:10.911492 24601 raft_consensus.cc:493] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.912330 24601 raft_consensus.cc:515] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.913379 24364 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.917899 24455 raft_consensus.cc:3055] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:10.918440 24455 raft_consensus.cc:740] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:10.919962 24572 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251214 03:00:10.920702 24616 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:10.921777 24485 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.921516 24616 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.923127 24595 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.924069 24595 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.906978 24609 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.922554 24485 raft_consensus.cc:515] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:10.903896 21275 rpcz_store.cc:269] 1214 03:00:07.895425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:07.895731 (+ 306us) service_pool.cc:225] Handling call
1214 03:00:10.903008 (+3007277us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001fccd4. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670efa1f7a 00007f6713f53647 00007f670d4e3834 00007f670d4e37c2 00007f670d4e36f1 00007f670d4e366c 00007f670d4e2922 00007f670d2dd55c 00007f670d2dc996 00007f670d2dc33f
1214 03:00:10.903195 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2359078144}
I20251214 03:00:10.936851 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47062 (request call id 2192) took 2423 ms. Trace:
I20251214 03:00:10.938707 24370 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:10.939595 24370 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.937348 21133 rpcz_store.cc:276] 1214 03:00:08.513106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.513239 (+ 133us) service_pool.cc:225] Handling call
1214 03:00:08.513538 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:08.513741 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 7
1214 03:00:10.777619 (+2263878us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b1800323980. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f6714d41f21 00007f6714d4e901 00007f670eb4585c 00007f670bed2ffb 00007f6714cba7e9 00007f670beef28b 00007f670beed52f 00007f670beed045 00007f6712998317 00007f670efa8627 00007f670efb7d3f
1214 03:00:10.778008 (+ 389us) log.cc:844] Serialized 14 byte log entry
1214 03:00:10.936052 (+158044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:10.936153 (+ 101us) raft_consensus.cc:1645] Marking committed up to 7
1214 03:00:10.936237 (+ 84us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:10.936367 (+ 130us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:10.936602 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51389,"prepare.queue_time_us":84777,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":1107,"raft.queue_time_us":51811,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":272,"spinlock_wait_cycles":7301504,"thread_start_us":185016,"threads_started":2}
I20251214 03:00:10.941984 24370 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.910279 24464 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.890025 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:10.943338 24464 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:10.948714 24464 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.934154 24609 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.017580 24548 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
W20251214 03:00:11.017473 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 151974us
I20251214 03:00:10.908638 24397 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:10.934219 24595 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:10.932278 24578 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.018059 24538 raft_consensus.cc:2804] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:10.949559 24573 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.996573 24537 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.031801 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.033110 24538 raft_consensus.cc:493] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.037463 24545 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:10.948734 24485 leader_election.cc:290] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.968745 24401 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.037421 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:10.968152 24400 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.019385 24548 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.052431 22014 raft_consensus.cc:1275] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 03:00:11.053436 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.054441 24548 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.054946 21130 raft_consensus.cc:1240] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:11.055773 24571 leader_election.cc:290] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.057446 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.058365 22013 raft_consensus.cc:2393] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:11.043454 21171 leader_election.cc:400] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 03:00:11.058759 21171 leader_election.cc:403] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251214 03:00:11.059695 24465 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Advancing to term 6
I20251214 03:00:10.903924 24617 raft_consensus.cc:515] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.047586 24601 leader_election.cc:290] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:10.972687 24407 raft_consensus.cc:493] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.061075 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.059965 24444 leader_election.cc:290] T 278f5a62c6a74101ae76cfc7325bcca0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.048835 24538 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:11.047233 24545 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:10.998456 24621 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20251214 03:00:11.048273 24397 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.061254 24609 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.070007 22011 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:10.920039 24455 consensus_queue.cc:260] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.073160 24264 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.073884 21133 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.074616 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.074898 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.075372 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:11.076115 24545 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.091408 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.091513 24264 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.092978 21282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 578) took 1886 ms. Trace:
I20251214 03:00:11.093494 24616 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.095953 21199 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 9: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:11.077054 24495 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:11.099355 24465 raft_consensus.cc:2749] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395) with higher term. Message: Invalid argument: T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6.
I20251214 03:00:11.099195 24495 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.101430 24495 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.078462 24400 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.080777 24454 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:11.081899 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:11.087664 24617 leader_election.cc:290] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.090969 21202 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:11.104099 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.108104 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.109472 21124 raft_consensus.cc:1275] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 03:00:11.105798 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.107419 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.103107 21171 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:11.108778 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.119052 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.119246 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.119640 21279 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 03:00:11.118126 24444 raft_consensus.cc:493] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.120074 21126 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 13: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.119807 21273 raft_consensus.cc:2468] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 03:00:11.102516 21206 consensus_meta.cc:287] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.532s user 0.005s sys 0.005s
I20251214 03:00:11.119834 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.117961 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.123921 21274 raft_consensus.cc:2393] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.124070 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:11.117954 21276 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.123994 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.127269 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.127923 24534 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.129729 21133 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.124308 24444 raft_consensus.cc:515] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.119987 24566 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.131763 24566 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.133785 24566 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.134227 21275 raft_consensus.cc:2393] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.128450 24534 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.137468 21202 raft_consensus.cc:2468] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 11.
I20251214 03:00:11.138484 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.139284 24444 leader_election.cc:290] T 19fd2760f37a43e2983b015db150e285 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.140193 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.140879 21274 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 03:00:11.120782 24454 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:11.141875 24557 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 1.969s user 0.000s sys 0.019s
I20251214 03:00:11.143438 21205 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 03:00:11.128376 21099 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:11.144510 21201 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20251214 03:00:11.129102 21206 raft_consensus.cc:2468] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 13.
W20251214 03:00:11.149031 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 46190.3us
I20251214 03:00:11.150233 21171 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:11.151067 21130 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:11.151862 24444 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:11.128587 24511 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20251214 03:00:11.152612 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0e59ca14c1548a686e85b1a9fcf9288" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.153609 24534 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.154970 21203 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42032 (request call id 1707) took 2489 ms. Trace:
I20251214 03:00:11.155532 24454 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.156170 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.157335 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:11.157018 21027 catalog_manager.cc:5654] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 668ad4f30bd649c88a034bbab9f13b20 reported cstate change: term changed from 3 to 5, leader changed from 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68) to 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66). New cstate: current_term: 5 leader_uuid: "668ad4f30bd649c88a034bbab9f13b20" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 03:00:11.158106 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.158907 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.159056 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:11.155342 21203 rpcz_store.cc:276] 1214 03:00:08.665678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.665792 (+ 114us) service_pool.cc:225] Handling call
1214 03:00:08.666078 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:08.666393 (+ 315us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:08.666420 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:08.666855 (+ 435us) log.cc:844] Serialized 26 byte log entry
1214 03:00:08.845136 (+178281us) raft_consensus.cc:1645] Marking committed up to 7
1214 03:00:08.845180 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:08.845252 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:11.141058 (+2295806us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 03:00:11.154530 (+ 13472us) raft_consensus.cc:1685] finished
1214 03:00:11.154550 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:11.154791 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2304,"raft.queue_time_us":740,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":93440,"spinlock_wait_cycles":361453312,"thread_start_us":175632,"threads_started":1,"wal-append.queue_time_us":363103}
I20251214 03:00:11.152796 22013 raft_consensus.cc:2393] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 4: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:11.160109 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.159739 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.164458 21132 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 13: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.164742 21126 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.153911 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.164901 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.164487 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.094691 21282 rpcz_store.cc:276] 1214 03:00:09.182052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:09.182178 (+ 126us) service_pool.cc:225] Handling call
1214 03:00:09.182472 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:09.182716 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 8
1214 03:00:10.487042 (+1304326us) log.cc:844] Serialized 14 byte log entry
1214 03:00:11.030319 (+543277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1214 03:00:11.044722 (+ 14403us) raft_consensus.cc:1645] Marking committed up to 8
1214 03:00:11.048997 (+ 4275us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.061802 (+ 12805us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:11.068683 (+ 6881us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9289,"prepare.queue_time_us":1461810,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":21628,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"spinlock_wait_cycles":6996224,"thread_start_us":1684513,"threads_started":2,"wal-append.queue_time_us":536316}
I20251214 03:00:11.159713 24400 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.164228 21129 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
I20251214 03:00:11.170220 21124 raft_consensus.cc:2468] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 03:00:11.170550 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.171216 21127 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:11.171733 21133 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 11.
I20251214 03:00:11.172062 24567 raft_consensus.cc:2804] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:11.158139 24590 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.136319 24397 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.172603 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.205886 21204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 3303) took 2392 ms. Trace:
I20251214 03:00:11.182925 21128 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 03:00:11.205185 22011 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 7.
W20251214 03:00:11.208133 24626 log.cc:927] Time spent T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.125s user 0.002s sys 0.000s
I20251214 03:00:11.122998 24407 raft_consensus.cc:515] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.208688 22018 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:11.213477 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:11.182943 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.183523 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.183495 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.210100 21204 rpcz_store.cc:276] 1214 03:00:08.813219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.818652 (+ 5433us) service_pool.cc:225] Handling call
1214 03:00:08.818908 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:09.916636 (+1097728us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.205400 (+1288764us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b54001cb454. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 03:00:11.205692 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1721842432}
I20251214 03:00:11.178592 24566 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.172782 24444 raft_consensus.cc:2804] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:11.179018 24455 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:11.237463 24571 raft_consensus.cc:493] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:11.238617 21247 connection.cc:232] RPC call timeout handler was delayed by -1.0656826840000804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20251214 03:00:11.239095 24609 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.239730 24580 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.240362 24576 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.249284 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.247164 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:11.271082 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.240933 24459 raft_consensus.cc:493] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.241770 21280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 583) took 1897 ms. Trace:
W20251214 03:00:11.240460 24606 log.cc:927] Time spent T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.538s user 0.002s sys 0.000s
I20251214 03:00:11.242182 24616 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.273031 24616 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.243403 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 600) took 2325 ms. Trace:
I20251214 03:00:11.244858 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:11.275205 24571 raft_consensus.cc:515] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.278126 24571 leader_election.cc:290] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.272023 24459 raft_consensus.cc:515] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.279517 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.279844 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.280778 24459 leader_election.cc:290] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.281013 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.236757 24444 raft_consensus.cc:493] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.280139 21124 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:11.246522 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.253774 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.255718 24481 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.256834 24538 raft_consensus.cc:515] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:11.259615 24627 log.cc:927] Time spent T 172b9f2bc43b42fca62c756aa5cf17a1 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.194s user 0.002s sys 0.000s
W20251214 03:00:11.286517 21247 connection.cc:232] RPC call timeout handler was delayed by -1.0494746130000294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
I20251214 03:00:11.287997 24538 leader_election.cc:290] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.257596 24550 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.288295 24585 raft_consensus.cc:493] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.288707 24584 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.288865 24585 raft_consensus.cc:515] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:11.289410 21247 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.710s (SENT) [suppressed 17 similar messages]
I20251214 03:00:11.289319 24584 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.258750 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.248518 22012 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "172b9f2bc43b42fca62c756aa5cf17a1" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.263228 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.251650 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.248224 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.251817 21200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47944 (request call id 601) took 2314 ms. Trace:
W20251214 03:00:11.259578 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 80655.8us
I20251214 03:00:11.269440 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.258316 24420 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:11.270141 24576 raft_consensus.cc:515] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.244104 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.271997 24609 raft_consensus.cc:515] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.246430 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.281634 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.282384 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.245715 24397 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.280105 21281 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:11.172626 24567 raft_consensus.cc:697] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:11.258071 21276 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 03:00:11.290729 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 55150.4us
I20251214 03:00:11.294296 24420 raft_consensus.cc:493] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.291638 21200 rpcz_store.cc:276] 1214 03:00:08.937698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.937824 (+ 126us) service_pool.cc:225] Handling call
1214 03:00:08.938108 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
1214 03:00:10.589827 (+1651719us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.251384 (+661557us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540024b654. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670ef075a1 00007f670ef9ea81 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22 00007f670d2dd55c 00007f670d2dc996
1214 03:00:11.251655 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3794781568}
I20251214 03:00:11.294996 21274 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 03:00:11.295372 24585 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.372294 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.372793 24576 leader_election.cc:290] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.373046 21203 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:11.274778 21207 rpcz_store.cc:276] 1214 03:00:08.918165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.932124 (+ 13959us) service_pool.cc:225] Handling call
1214 03:00:08.932490 (+ 366us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:10.572551 (+1640061us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:10.572582 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:10.591107 (+ 18525us) log.cc:844] Serialized 26 byte log entry
1214 03:00:11.065577 (+474470us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:11.067332 (+ 1755us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.071415 (+ 4083us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:11.207362 (+135947us) spinlock_profiling.cc:243] Waited 7.04 ms on lock 0x7b440002304c. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670bd30781 00007f670bdfeed5 00007f670bdff8f2 00007f670eb4957b 00007f670eb44e4f 00007f670eb38c0e 00007f670eb37f56 00007f670eb37b78 00007f670eb415c1 00007f670eb41579 00007f670eb41509
1214 03:00:11.242931 (+ 35569us) raft_consensus.cc:1685] finished
1214 03:00:11.242958 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:11.243229 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":164860,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":33172,"spinlock_wait_cycles":16190208,"thread_start_us":1819985,"threads_started":2,"wal-append.queue_time_us":442436}
I20251214 03:00:11.292012 21279 raft_consensus.cc:2468] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
W20251214 03:00:11.360157 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 70760.7us
W20251214 03:00:11.375543 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 91207.2us
I20251214 03:00:11.376932 22017 raft_consensus.cc:2468] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 5.
I20251214 03:00:11.294493 22011 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:11.272149 21280 rpcz_store.cc:276] 1214 03:00:09.344731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:09.344875 (+ 144us) service_pool.cc:225] Handling call
1214 03:00:09.425939 (+ 81064us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:09.579796 (+153857us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:09.579825 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:10.906363 (+1326538us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b2c00107c08. stack: 00007f670bebe083 00007f670bebdf80 00007f670b810574 00007f67154cb6b4 00007f67154cb649 00007f670d89c9b1 00007f670d8abfe2 00007f670efd9851 00007f670ef9e0ba 00007f670ef9d4be 00007f6713f51f49 00007f670d4ded34 00007f670d4decc2 00007f670d4debf1 00007f670d4deb6c 00007f670d4dde22
1214 03:00:11.003232 (+ 96869us) log.cc:844] Serialized 26 byte log entry
1214 03:00:11.050162 (+ 46930us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:11.050201 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.063320 (+ 13119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:11.241296 (+177976us) raft_consensus.cc:1685] finished
1214 03:00:11.241314 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:11.241603 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":141207,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4725,"spinlock_wait_cycles":16021248,"thread_start_us":199812,"threads_started":2,"wal-append.queue_time_us":86442}
I20251214 03:00:11.383303 21129 raft_consensus.cc:2393] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
W20251214 03:00:11.383796 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:11.310878 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.375569 21125 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:11.249215 24264 leader_election.cc:290] T ae2c1119874240b6af01af115f82ecd3 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.284770 24444 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:11.391075 22018 raft_consensus.cc:1240] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:11.391628 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.395198 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.398205 24436 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:11.398891 24436 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.400967 24436 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.401199 21127 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 03:00:11.402161 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.402740 24489 raft_consensus.cc:493] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:11.403429 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.403622 21282 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 03:00:11.404192 21127 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:11.404835 21986 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:11.405043 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.405718 21127 raft_consensus.cc:2468] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 12.
I20251214 03:00:11.406085 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.406859 21203 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 5.
I20251214 03:00:11.407146 21129 raft_consensus.cc:1275] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 03:00:11.402256 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.397030 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.296078 24601 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.376570 24420 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:11.274923 24580 raft_consensus.cc:515] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.412951 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.412878 21126 raft_consensus.cc:2468] T 75e354ca56944ff1b7775a8309bca568 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 4.
I20251214 03:00:11.417732 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.417994 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.418498 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.417752 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.423616 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.424090 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.313448 24482 raft_consensus.cc:493] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.428474 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.432392 21986 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:11.288733 24550 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.437992 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.310807 24566 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:11.382251 24485 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.440171 24616 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.438484 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.440109 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.444702 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.440833 24436 consensus_queue.cc:1059] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251214 03:00:11.447536 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.447875 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1dc028308e3434791fdd7169a0858b0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:11.449107 24613 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:11.313752 24400 consensus_queue.cc:1048] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.123s
I20251214 03:00:11.437589 24454 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:11.419473 24264 raft_consensus.cc:493] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.403453 24489 raft_consensus.cc:515] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.452385 24584 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.550925 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.551326 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.556247 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.494220 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
W20251214 03:00:11.492027 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 46119.6us
W20251214 03:00:11.536013 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:11.667117 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.519380 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.533684 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:11.714912 24609 leader_election.cc:290] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.493224 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:11.731434 21281 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
W20251214 03:00:11.520830 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 52470.4us
I20251214 03:00:11.798159 21275 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 10: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:11.764088 21282 raft_consensus.cc:2468] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:11.764726 21279 raft_consensus.cc:2468] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:11.802412 21126 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
W20251214 03:00:11.495348 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 52268.4us
I20251214 03:00:11.441200 24614 consensus_queue.cc:1059] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.392s
I20251214 03:00:11.806864 21245 leader_election.cc:304] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:11.547717 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.545670 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.808036 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.809537 21278 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 03:00:11.810066 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.807423 24614 raft_consensus.cc:3055] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 LEADER]: Stepping down as leader of term 10
I20251214 03:00:11.806870 24264 raft_consensus.cc:515] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.823036 24550 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.823292 21099 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:11.516806 24601 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:11.646443 24634 log.cc:927] Time spent T 75c047f1d6d94237afc050db2241d659 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20251214 03:00:11.759949 24567 consensus_queue.cc:237] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.816239 21273 raft_consensus.cc:2468] T 205cf8366aa44b27a349916da03cff46 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 03:00:11.762732 24454 raft_consensus.cc:493] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:11.845638 24549 raft_consensus.cc:2804] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 03:00:11.542692 24533 consensus_queue.cc:1048] T 5751534b61ed4568abd62057238d5fda P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.335s
I20251214 03:00:11.836557 24584 raft_consensus.cc:2804] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:11.500895 24407 leader_election.cc:290] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.444649 24481 raft_consensus.cc:493] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:11.757642 24613 raft_consensus.cc:493] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:11.845397 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 54311.6us
I20251214 03:00:11.857291 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.861702 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.827507 24614 raft_consensus.cc:740] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:11.520179 24482 raft_consensus.cc:515] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:11.856186 24454 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:11.879946 21277 raft_consensus.cc:2468] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 03:00:11.884800 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.841071 24485 raft_consensus.cc:515] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:11.856005 24584 raft_consensus.cc:493] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:11.892676 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.899580 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:11.884831 24549 raft_consensus.cc:697] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:11.908653 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.909176 21099 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:11.883577 24613 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:11.900004 24584 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:11.922997 21274 raft_consensus.cc:3060] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:11.924214 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.924172 21277 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:11.924880 21282 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 6.
I20251214 03:00:11.899324 24580 leader_election.cc:290] T 755b818717824383985748e28244d7a1 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:11.934996 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.945801 21273 raft_consensus.cc:1275] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251214 03:00:11.928558 21096 leader_election.cc:304] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:11.922791 24601 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
W20251214 03:00:11.955014 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:11.934233 24566 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:11.958608 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:11.960435 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:11.959367 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 299408us
I20251214 03:00:11.961979 21282 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 03:00:11.941324 24533 raft_consensus.cc:2804] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 13 FOLLOWER]: Leader election won for term 13
I20251214 03:00:11.990988 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.994700 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.995352 21277 raft_consensus.cc:2468] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 3.
I20251214 03:00:11.972154 24489 leader_election.cc:290] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:11.997773 24361 consensus_queue.cc:1048] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20251214 03:00:11.991055 24533 raft_consensus.cc:697] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [term 13 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.011897 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.993450 24535 raft_consensus.cc:2749] T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:12.013289 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:12.024488 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 103333us
W20251214 03:00:12.010068 24420 consensus_meta.cc:287] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 0.560s user 0.011s sys 0.000s
I20251214 03:00:12.029336 24533 consensus_queue.cc:237] T a54bd91b04344ee0b499fb117cc9fb91 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.031342 24482 leader_election.cc:290] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.031962 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.032436 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.024291 24549 consensus_queue.cc:237] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.033349 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.033504 21281 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 03:00:12.034018 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.034152 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.034924 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.035030 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:11.961314 24614 consensus_queue.cc:260] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:12.035619 24641 log.cc:927] Time spent T ae2c1119874240b6af01af115f82ecd3 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20251214 03:00:12.034941 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.053455 21132 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 03:00:12.039160 24485 leader_election.cc:290] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.046590 21274 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:12.046828 21128 raft_consensus.cc:2393] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 10: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.047768 21096 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:12.054301 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "278f5a62c6a74101ae76cfc7325bcca0" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:11.919586 24481 raft_consensus.cc:515] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.055027 21126 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 03:00:12.064985 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.039795 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:12.057715 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 205505us
I20251214 03:00:12.067672 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:12.062047 24614 raft_consensus.cc:3060] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Advancing to term 11
I20251214 03:00:12.081347 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.081995 21133 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 9.
I20251214 03:00:12.057611 24264 leader_election.cc:290] T 96ca056dd7594a23a70918fc54c5eb2a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:12.085201 24643 log.cc:927] Time spent T 172b9f2bc43b42fca62c756aa5cf17a1 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20251214 03:00:12.087847 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.087150 21129 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 03:00:12.090075 22020 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.093715 21202 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20251214 03:00:12.094166 21279 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [term 6 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:12.053283 24420 raft_consensus.cc:515] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.097286 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.098009 22013 raft_consensus.cc:2393] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.079919 24572 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:12.093034 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.102834 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.103353 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.103897 21274 raft_consensus.cc:2468] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 11.
I20251214 03:00:12.105850 24471 raft_consensus.cc:2804] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:12.089659 24397 raft_consensus.cc:3055] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:12.118541 21199 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 03:00:12.121326 21205 raft_consensus.cc:2393] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:12.121912 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.123323 21279 raft_consensus.cc:2468] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 5.
I20251214 03:00:12.102587 24572 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 03:00:12.119695 24638 log.cc:927] Time spent T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20251214 03:00:12.106253 24471 raft_consensus.cc:493] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:12.133385 21279 raft_consensus.cc:1275] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 03:00:12.117342 24397 raft_consensus.cc:740] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:12.141817 21280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.145046 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.132072 24572 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:12.155141 24515 consensus_queue.cc:1059] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
W20251214 03:00:12.156939 24645 log.cc:927] Time spent T 4b611d0c068c451bb4eeb7eda4823036 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20251214 03:00:12.136194 24471 raft_consensus.cc:3060] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:12.158986 21280 raft_consensus.cc:2410] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20251214 03:00:12.163391 21203 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:12.159852 24361 consensus_queue.cc:1059] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
W20251214 03:00:12.168742 21174 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.174070 21282 raft_consensus.cc:1275] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 03:00:12.175194 24485 consensus_queue.cc:1048] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20251214 03:00:12.182847 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
W20251214 03:00:12.185474 21984 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.197650 21279 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 13: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:12.202318 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.206364 21280 raft_consensus.cc:3060] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.217356 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:12.221648 22015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.222368 22015 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.207226 24481 leader_election.cc:290] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:12.228084 21279 raft_consensus.cc:1275] T 639e38cdc0a74102bbcd86211abe87a0 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 03:00:12.230691 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:12.213896 24420 leader_election.cc:290] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.232642 21203 raft_consensus.cc:2393] T a54bd91b04344ee0b499fb117cc9fb91 P 668ad4f30bd649c88a034bbab9f13b20 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 13: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.238098 21207 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.231499 21247 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.239331 21205 raft_consensus.cc:1275] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251214 03:00:12.247906 22015 raft_consensus.cc:2468] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:12.252913 21171 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 03:00:12.252074 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 246075us
I20251214 03:00:12.257890 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.263473 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:12.205430 24397 consensus_queue.cc:260] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:12.255425 21098 connection.cc:232] RPC call timeout handler was delayed by -2.1929537209999808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
I20251214 03:00:12.269510 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.274860 21205 raft_consensus.cc:1275] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Refusing update from remote peer 6e36caefa37d47a5968cffd9e270044f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251214 03:00:12.276135 21282 raft_consensus.cc:2393] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 721d604130924d939d291fe3eef8f6d1 in current term 9: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:12.255980 24420 raft_consensus.cc:2804] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:12.287477 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.274891 24576 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:12.293864 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:12.292572 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 61130.5us
I20251214 03:00:12.298045 21247 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.300942 21203 raft_consensus.cc:2393] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:12.294778 24397 raft_consensus.cc:3060] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.305034 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.298835 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.307956 21200 raft_consensus.cc:2393] T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.290889 24420 raft_consensus.cc:697] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:12.296033 24576 raft_consensus.cc:493] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:12.307170 21207 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:12.313395 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.306428 24485 raft_consensus.cc:2804] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251214 03:00:12.310482 24455 consensus_meta.cc:287] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 1.026s user 0.009s sys 0.004s
I20251214 03:00:12.315822 21278 raft_consensus.cc:2468] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 9.
I20251214 03:00:12.319836 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.320415 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19fd2760f37a43e2983b015db150e285" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.312520 24576 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.325266 21247 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:12.330070 21984 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.331475 21200 raft_consensus.cc:2468] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:12.342204 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.321659 24485 raft_consensus.cc:493] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:12.340533 24482 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.354245 21247 leader_election.cc:304] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 03:00:12.383693 21098 connection.cc:232] RPC call timeout handler was delayed by -2.0341238529999828s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
I20251214 03:00:12.347818 24485 raft_consensus.cc:3060] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:12.355538 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:12.384929 21098 connection.cc:232] RPC call timeout handler was delayed by -2.0327537209999491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17390000000000003s for a response
W20251214 03:00:12.354967 24642 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20251214 03:00:12.394281 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0e04af2e6054ecabda3186cef55253f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.388963 21207 raft_consensus.cc:2468] T 275e8242241b4fd386a7c5c738a2c903 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
W20251214 03:00:12.399650 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 57827.3us
I20251214 03:00:12.400306 21205 raft_consensus.cc:2468] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
W20251214 03:00:12.400892 21098 connection.cc:232] RPC call timeout handler was delayed by -1.6655537209999238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2147s for a response
W20251214 03:00:12.401286 21098 connection.cc:232] RPC call timeout handler was delayed by -1.2545727549999128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
W20251214 03:00:12.401618 21098 connection.cc:232] RPC call timeout handler was delayed by -1.1681727549998868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
I20251214 03:00:12.403505 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.408399 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.409065 21247 leader_election.cc:304] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.349187 24584 raft_consensus.cc:515] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.411263 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.388962 24482 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:12.389473 24654 raft_consensus.cc:493] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 03:00:12.412799 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 128638us
I20251214 03:00:12.409904 21199 raft_consensus.cc:2468] T 5bdfefe7cfb64f5cba716a427e83fc9f P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 11.
I20251214 03:00:12.417181 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.394580 24481 raft_consensus.cc:493] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
W20251214 03:00:12.408478 24454 consensus_meta.cc:287] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.503s user 0.007s sys 0.004s
I20251214 03:00:12.421057 21124 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 9: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:12.403990 24616 raft_consensus.cc:2804] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20251214 03:00:12.416285 24650 log.cc:927] Time spent T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20251214 03:00:12.429425 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.430754 21277 raft_consensus.cc:1240] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:12.434048 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75c047f1d6d94237afc050db2241d659" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.434695 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320811ca9d184da38a1197ebfdb7cd0f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.435103 21207 raft_consensus.cc:1240] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 9. Current term is 10. Ops: []
I20251214 03:00:12.400385 24533 consensus_queue.cc:1059] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20251214 03:00:12.438340 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.438263 21247 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.417863 24482 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.421224 24489 raft_consensus.cc:2804] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:12.441880 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.444212 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:12.451372 24651 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20251214 03:00:12.455118 24567 raft_consensus.cc:3055] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Stepping down as leader of term 3
I20251214 03:00:12.455549 24567 raft_consensus.cc:740] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.456336 24567 consensus_queue.cc:260] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.459148 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 3333) took 1033 ms. Trace:
I20251214 03:00:12.429796 24616 raft_consensus.cc:493] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:12.442974 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ce7181915c14c369bac4b962d34913a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.465889 21126 raft_consensus.cc:1240] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 10. Current term is 11. Ops: []
I20251214 03:00:12.449973 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.468712 21247 leader_election.cc:304] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.464237 24601 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:12.464843 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.471508 24601 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:12.471860 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf5cb5c0f4854e4a846c25eb5ce80a9a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.472599 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.403196 24420 consensus_queue.cc:237] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.459496 21201 rpcz_store.cc:276] 1214 03:00:11.425683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:11.425831 (+ 148us) service_pool.cc:225] Handling call
1214 03:00:11.440121 (+ 14290us) raft_consensus.cc:1471] Updating replica for 1 ops
1214 03:00:11.449962 (+ 9841us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 03:00:11.450000 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1214 03:00:11.450524 (+ 524us) log.cc:844] Serialized 26 byte log entry
1214 03:00:11.765786 (+315262us) raft_consensus.cc:1645] Marking committed up to 6
1214 03:00:11.765826 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1214 03:00:11.765891 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1214 03:00:12.458972 (+693081us) raft_consensus.cc:1685] finished
1214 03:00:12.458991 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1214 03:00:12.459098 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4661,"raft.queue_time_us":10240,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":145,"thread_start_us":314876,"threads_started":1,"wal-append.queue_time_us":749173}
I20251214 03:00:12.482952 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:12.458997 24444 consensus_meta.cc:287] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20: Time spent flushing consensus metadata: real 1.026s user 0.013s sys 0.001s
I20251214 03:00:12.428210 24454 raft_consensus.cc:515] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.493928 21207 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:12.494208 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:12.494702 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.419872 24654 raft_consensus.cc:515] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.448693 24489 raft_consensus.cc:493] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:12.500561 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.471935 24601 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:12.465159 24616 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:12.459601 24657 raft_consensus.cc:493] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:12.505487 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.509595 21173 leader_election.cc:304] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.514330 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.514683 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.518201 21200 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.519165 21207 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
W20251214 03:00:12.520931 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.521142 21199 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.521901 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.458950 24481 raft_consensus.cc:515] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.522583 21202 raft_consensus.cc:2468] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 11.
I20251214 03:00:12.522850 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.518600 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.523169 22017 raft_consensus.cc:1275] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20251214 03:00:12.506106 24489 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:12.527269 21207 raft_consensus.cc:2393] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:12.531008 24666 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:12.535148 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.535431 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae2c1119874240b6af01af115f82ecd3" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.538400 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.539242 21208 raft_consensus.cc:2468] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:12.537808 21204 raft_consensus.cc:2468] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 9.
I20251214 03:00:12.498555 24444 raft_consensus.cc:515] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:12.544189 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 75566.4us
I20251214 03:00:12.547894 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b611d0c068c451bb4eeb7eda4823036" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.545670 21200 raft_consensus.cc:2468] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 5.
I20251214 03:00:12.550318 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.550953 21125 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 8.
W20251214 03:00:12.549921 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 130869us
I20251214 03:00:12.544744 24538 raft_consensus.cc:2804] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.559222 21985 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.560112 21247 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:12.562091 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.563050 21248 leader_election.cc:304] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:12.559654 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.566300 21279 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 03:00:12.566426 21203 raft_consensus.cc:1240] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 7. Current term is 8. Ops: []
I20251214 03:00:12.567520 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.568174 21279 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:12.572543 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.574815 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:12.574518 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.519868 24657 raft_consensus.cc:515] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.579805 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ce8bd94962e4bf48e01e9e747adba15" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.580883 21985 leader_election.cc:304] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.580503 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.581970 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:12.577623 21173 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.563868 24538 raft_consensus.cc:493] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.587445 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:12.569229 24584 leader_election.cc:290] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.590082 21279 raft_consensus.cc:1240] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20251214 03:00:12.581086 24615 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:12.595052 24481 leader_election.cc:290] T d55cde2542d544c5b99e6113b95910b8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:12.587299 24613 consensus_meta.cc:287] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.654s user 0.009s sys 0.003s
I20251214 03:00:12.589572 24538 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.602252 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.602161 24456 raft_consensus.cc:2749] T 6ae31f8b580840eb863e4d400367c1be P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:12.607241 21985 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.601620 24613 raft_consensus.cc:515] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.612633 21173 leader_election.cc:304] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.618180 24666 raft_consensus.cc:2749] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:12.622007 21173 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.629945 24566 raft_consensus.cc:2804] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:12.630182 24613 leader_election.cc:290] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.601416 24584 raft_consensus.cc:493] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:12.600740 24481 raft_consensus.cc:2749] T e18f39bed60f44f5b8c25ede5d8bba8f P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:12.634667 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:12.635326 21173 leader_election.cc:304] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
W20251214 03:00:12.645509 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 73055.8us
I20251214 03:00:12.653812 21173 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.656935 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:12.657536 21203 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:12.658510 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 95630.2us
I20251214 03:00:12.627923 24667 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.642671 24566 raft_consensus.cc:493] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.633776 24436 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:12.662407 24364 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 668ad4f30bd649c88a034bbab9f13b20 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:12.634182 24454 leader_election.cc:290] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.667709 21098 leader_election.cc:304] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:12.667726 24663 log.cc:927] Time spent T 172b9f2bc43b42fca62c756aa5cf17a1 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20251214 03:00:12.669754 24482 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.708343 24482 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.671825 21248 leader_election.cc:304] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:12.709491 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 47314.6us
I20251214 03:00:12.710520 24677 raft_consensus.cc:2749] T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:12.711179 24677 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:12.706461 24616 raft_consensus.cc:515] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } }
I20251214 03:00:12.711750 21248 leader_election.cc:304] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.713250 21248 leader_election.cc:304] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.713380 24616 leader_election.cc:290] T 2734ae274c16411d93b2c95d7fb3e00c P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.644815 24264 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251214 03:00:12.660624 24667 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.695307 24657 leader_election.cc:290] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.695822 24444 leader_election.cc:290] T 6ae31f8b580840eb863e4d400367c1be P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.660565 24654 leader_election.cc:290] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.647413 24584 raft_consensus.cc:515] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.697566 24489 raft_consensus.cc:515] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.697743 24601 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:12.699851 24614 consensus_meta.cc:287] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Time spent flushing consensus metadata: real 0.621s user 0.004s sys 0.008s
I20251214 03:00:12.700433 21203 raft_consensus.cc:2468] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 03:00:12.714371 21273 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 9. Current term is 10. Ops: []
W20251214 03:00:12.700675 21096 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 118486us
W20251214 03:00:12.691430 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 131394us
I20251214 03:00:12.736122 21986 leader_election.cc:304] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:12.662014 24666 raft_consensus.cc:2804] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:12.671353 24481 raft_consensus.cc:2804] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.683701 24566 raft_consensus.cc:3060] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:12.723565 21245 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 170350us
I20251214 03:00:12.724431 21985 leader_election.cc:304] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.724490 24482 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20251214 03:00:12.725323 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.726854 24576 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.727727 24671 raft_consensus.cc:2749] T c8ab1ed25694443cb4efebfc35e5205e P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 03:00:12.697137 24471 consensus_meta.cc:287] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.526s user 0.008s sys 0.000s
I20251214 03:00:12.764159 22018 raft_consensus.cc:1240] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
W20251214 03:00:12.727676 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 60077.8us
I20251214 03:00:12.733374 24672 raft_consensus.cc:2749] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251214 03:00:12.724387 24667 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.769642 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.730125 24616 raft_consensus.cc:2749] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20251214 03:00:12.743324 24561 raft_consensus.cc:2804] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:12.743810 24613 raft_consensus.cc:2749] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:12.744846 21985 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:12.778506 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 189751us
I20251214 03:00:12.745308 24584 leader_election.cc:290] T e91314b7fb3c4dbd8db2eebcf16d820f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.746762 24601 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.729979 24538 raft_consensus.cc:515] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:12.752140 24572 consensus_meta.cc:287] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.585s user 0.006s sys 0.003s
I20251214 03:00:12.752974 21247 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.739396 24666 raft_consensus.cc:493] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.788192 24666 raft_consensus.cc:3060] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.788293 21247 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.789443 24672 raft_consensus.cc:2804] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20251214 03:00:12.789785 24672 raft_consensus.cc:493] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:12.790148 24672 raft_consensus.cc:3060] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Advancing to term 13
I20251214 03:00:12.791210 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.791934 22018 raft_consensus.cc:2393] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 7: Already voted for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in this term.
I20251214 03:00:12.791770 24482 consensus_queue.cc:1048] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251214 03:00:12.787734 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.794703 24584 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.795097 24584 raft_consensus.cc:493] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.795399 24584 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.741199 24481 raft_consensus.cc:493] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.798769 24481 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.771176 21278 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:12.752631 24654 consensus_queue.cc:1048] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20251214 03:00:12.805274 24485 raft_consensus.cc:515] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.773450 24576 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.768215 24471 raft_consensus.cc:515] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.807143 22017 raft_consensus.cc:1240] T 4ba4da4df7bc466aa9cc42fa304335d8 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 4 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 3. Current term is 4. Ops: []
I20251214 03:00:12.807866 24471 leader_election.cc:290] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.773669 24652 consensus_queue.cc:1059] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251214 03:00:12.776538 24659 raft_consensus.cc:3055] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:12.808578 24659 raft_consensus.cc:740] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.808693 24652 raft_consensus.cc:3055] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:12.809139 24652 raft_consensus.cc:740] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.809412 24659 consensus_queue.cc:260] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.787878 22011 raft_consensus.cc:1240] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:12.752501 24657 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:12.811245 21098 leader_election.cc:304] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:12.811148 24657 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.812641 21098 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:12.812834 22013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" is_pre_election: true
I20251214 03:00:12.813237 24657 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:12.781273 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:12.786367 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.790273 21247 leader_election.cc:304] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.784365 24456 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:12.787418 24572 raft_consensus.cc:515] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.787698 24561 raft_consensus.cc:697] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:12.798058 24672 raft_consensus.cc:515] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.816043 21247 leader_election.cc:304] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.806380 21276 raft_consensus.cc:1240] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 4. Current term is 5. Ops: []
I20251214 03:00:12.809870 24652 consensus_queue.cc:260] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.826853 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8cd37022763049b2829947b4a33d6aea" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:12.829080 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 115898us
I20251214 03:00:12.813817 24657 raft_consensus.cc:2804] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:12.810673 24659 raft_consensus.cc:3060] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:12.835619 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18f39bed60f44f5b8c25ede5d8bba8f" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:12.835999 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 182272us
I20251214 03:00:12.838327 21173 leader_election.cc:304] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.828676 24616 raft_consensus.cc:2804] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 03:00:12.845222 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fdf7633f868447a8abd2adfe9b04fc5" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.831132 24601 raft_consensus.cc:2749] T e1dc028308e3434791fdd7169a0858b0 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:12.824349 24613 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:12.854658 21098 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:12.824185 24682 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Leader election won for term 11
I20251214 03:00:12.828869 24361 consensus_queue.cc:1059] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20251214 03:00:12.865581 24613 raft_consensus.cc:493] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:12.871932 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:12.859781 24657 raft_consensus.cc:697] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.857609 24671 consensus_queue.cc:1059] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20251214 03:00:12.848657 24538 leader_election.cc:290] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:12.874172 21205 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.882164 21205 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 6.
I20251214 03:00:12.882521 21208 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20251214 03:00:12.847095 24616 raft_consensus.cc:493] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:12.883248 24616 raft_consensus.cc:3060] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Advancing to term 5
I20251214 03:00:12.850869 24561 consensus_queue.cc:237] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.880306 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4a2d7bee4345bbb9b7631ce9e5d5de" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:12.891254 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.878353 24584 raft_consensus.cc:515] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.881444 24471 raft_consensus.cc:2804] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election won for term 10
I20251214 03:00:12.836613 24666 raft_consensus.cc:515] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.875218 21203 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 03:00:12.848894 24652 raft_consensus.cc:3060] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:12.883898 24682 raft_consensus.cc:697] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.879462 21099 leader_election.cc:304] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.849897 24572 leader_election.cc:290] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.864423 21248 leader_election.cc:304] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.874763 24436 raft_consensus.cc:2749] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:12.869099 21173 leader_election.cc:304] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.889123 22019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd36aadbf4841c9a26e48e96430c009" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.901615 24584 leader_election.cc:290] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.891788 22017 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:12.890424 24670 consensus_queue.cc:1048] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20251214 03:00:12.853775 24481 raft_consensus.cc:515] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.880038 24613 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Advancing to term 9
I20251214 03:00:12.838495 24669 consensus_queue.cc:1059] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20251214 03:00:12.875906 24455 raft_consensus.cc:2749] T 143de581216646f79bece2b9f1259f64 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20251214 03:00:12.915128 21984 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.916002 24667 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.918416 24481 leader_election.cc:290] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:12.918910 24485 leader_election.cc:290] T c8ab1ed25694443cb4efebfc35e5205e P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:12.919535 24672 leader_election.cc:290] T 01028fc01cbf4f4c88f7e36696d08d03 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.907807 24682 consensus_queue.cc:237] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.921286 24657 consensus_queue.cc:237] T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.895713 24489 leader_election.cc:290] T 8c3be94c517348a08049e54abf8fd780 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:12.907234 24678 consensus_queue.cc:1048] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20251214 03:00:12.925311 24549 raft_consensus.cc:3055] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:12.925715 24549 raft_consensus.cc:740] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.926843 22017 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
I20251214 03:00:12.926494 24549 consensus_queue.cc:260] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:12.928949 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 59945.7us
W20251214 03:00:12.929771 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 51511.3us
W20251214 03:00:12.929153 21097 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:12.930135 21096 leader_election.cc:304] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
I20251214 03:00:12.930650 24361 raft_consensus.cc:2749] T 9fdf7633f868447a8abd2adfe9b04fc5 P 721d604130924d939d291fe3eef8f6d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 03:00:12.931243 24361 raft_consensus.cc:3055] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 03:00:12.931537 24361 raft_consensus.cc:740] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.931638 24649 raft_consensus.cc:2804] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election won for term 8
I20251214 03:00:12.931963 24471 raft_consensus.cc:697] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [term 10 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.932299 24361 consensus_queue.cc:260] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.933647 24649 raft_consensus.cc:697] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Becoming Leader. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:12.933462 24471 consensus_queue.cc:237] T e91314b7fb3c4dbd8db2eebcf16d820f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:12.934751 24649 consensus_queue.cc:237] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:12.933669 24361 raft_consensus.cc:3060] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.935220 21099 leader_election.cc:304] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:12.949913 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 85529.6us
I20251214 03:00:12.945101 21173 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.957449 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.962764 24670 raft_consensus.cc:2749] T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:12.963441 22019 raft_consensus.cc:1275] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20251214 03:00:12.963800 22019 pending_rounds.cc:85] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Aborting all ops after (but not including) 9
I20251214 03:00:12.967092 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:12.968506 22017 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:12.974002 21248 leader_election.cc:304] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:12.964008 22019 pending_rounds.cc:107] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20251214 03:00:12.975116 22019 raft_consensus.cc:2889] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 03:00:12.974387 21099 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1; no voters:
I20251214 03:00:12.976881 21099 leader_election.cc:304] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:12.956773 24455 raft_consensus.cc:2804] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:12.963950 24690 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 03:00:12.971304 24572 raft_consensus.cc:2749] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251214 03:00:12.970644 24667 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
W20251214 03:00:13.024827 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 44427us
I20251214 03:00:12.987293 24535 raft_consensus.cc:2749] T 9a4a2d7bee4345bbb9b7631ce9e5d5de P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:12.984608 24533 raft_consensus.cc:2804] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251214 03:00:13.024471 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 169911us
W20251214 03:00:13.030675 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 56747.9us
I20251214 03:00:12.987883 24455 raft_consensus.cc:493] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:12.971477 24566 raft_consensus.cc:515] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.019873 24670 raft_consensus.cc:2749] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:13.062525 22017 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 8de0cf3ca4bf4083bc494a9af0e49342 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 03:00:13.069722 21098 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:12.995786 24690 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 03:00:13.083877 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 43793.7us
I20251214 03:00:13.084748 21248 leader_election.cc:304] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:13.086094 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 76296.5us
I20251214 03:00:13.039453 24533 raft_consensus.cc:493] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6e36caefa37d47a5968cffd9e270044f)
I20251214 03:00:13.057713 24455 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
W20251214 03:00:13.090265 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 103582us
W20251214 03:00:13.092991 21171 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251214 03:00:13.102137 24660 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
I20251214 03:00:13.107642 21248 leader_election.cc:304] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.081297 24690 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:13.052552 24613 raft_consensus.cc:515] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
W20251214 03:00:13.112780 21098 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 52500.6us
I20251214 03:00:13.113523 24572 raft_consensus.cc:2749] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251214 03:00:13.093667 24535 consensus_queue.cc:1048] T f0e59ca14c1548a686e85b1a9fcf9288 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20251214 03:00:13.117398 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.118490 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143de581216646f79bece2b9f1259f64" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.118975 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f692dd0fa4b64450b9808a14da2b1c29" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.119352 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.119776 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.119925 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.123256 24566 leader_election.cc:290] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.125681 21027 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51648 (request call id 300) took 4844 ms. Trace:
I20251214 03:00:13.126096 21027 rpcz_store.cc:276] 1214 03:00:08.281710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 03:00:08.321952 (+ 40242us) service_pool.cc:225] Handling call
1214 03:00:13.125583 (+4803631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 03:00:12.136479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 03:00:12.286497 (+150018us) write_op.cc:432] Acquiring schema lock in shared mode
1214 03:00:12.293058 (+ 6561us) write_op.cc:435] Acquired schema lock
1214 03:00:12.298650 (+ 5592us) tablet.cc:660] Decoding operations
1214 03:00:12.343917 (+ 45267us) write_op.cc:620] Acquiring the partition lock for write op
1214 03:00:12.355788 (+ 11871us) write_op.cc:641] Partition lock acquired for write op
1214 03:00:12.382913 (+ 27125us) tablet.cc:683] Acquiring locks for 1 operations
1214 03:00:12.437815 (+ 54902us) tablet.cc:699] Row locks acquired
1214 03:00:12.442630 (+ 4815us) write_op.cc:260] PREPARE: finished
1214 03:00:12.462910 (+ 20280us) write_op.cc:270] Start()
1214 03:00:12.463100 (+ 190us) write_op.cc:276] Timestamp: P: 1765681212462844 usec, L: 0
1214 03:00:12.463145 (+ 45us) op_driver.cc:348] REPLICATION: starting
1214 03:00:12.463837 (+ 692us) log.cc:844] Serialized 536 byte log entry
1214 03:00:12.777994 (+314157us) op_driver.cc:464] REPLICATION: finished
1214 03:00:12.811706 (+ 33712us) write_op.cc:301] APPLY: starting
1214 03:00:12.811802 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
1214 03:00:12.811924 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
1214 03:00:12.812097 (+ 173us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 03:00:12.812422 (+ 325us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 03:00:12.812676 (+ 254us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 03:00:12.812718 (+ 42us) write_op.cc:312] APPLY: finished
1214 03:00:12.812990 (+ 272us) log.cc:844] Serialized 23 byte log entry
1214 03:00:12.828519 (+ 15529us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:12.828610 (+ 91us) write_op.cc:454] Released schema lock
1214 03:00:12.835321 (+ 6711us) write_op.cc:341] FINISH: Updating metrics
1214 03:00:12.862733 (+ 27412us) write_op.cc:489] Releasing partition, row and schema locks
1214 03:00:12.862764 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":133242,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":33452,"apply.run_cpu_time_us":2226,"apply.run_wall_time_us":45932,"num_ops":1,"prepare.queue_time_us":914431,"prepare.run_cpu_time_us":12789,"prepare.run_wall_time_us":393251,"raft.queue_time_us":144562,"raft.run_cpu_time_us":1805,"raft.run_wall_time_us":4854,"replication_time_us":314667,"thread_start_us":199445,"threads_started":4,"wal-append.queue_time_us":104294}]]}
I20251214 03:00:13.137194 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ae31f8b580840eb863e4d400367c1be" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.138237 21130 raft_consensus.cc:1240] T 3fd36aadbf4841c9a26e48e96430c009 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Rejecting Update request from peer 668ad4f30bd649c88a034bbab9f13b20 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:13.140326 24567 raft_consensus.cc:3060] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:13.142148 24549 raft_consensus.cc:3060] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:13.147743 24666 leader_election.cc:290] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.148306 22019 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 03:00:13.101528 24533 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:13.100107 24601 raft_consensus.cc:2749] T 19fd2760f37a43e2983b015db150e285 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20251214 03:00:13.155490 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 130645us
I20251214 03:00:13.169482 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.186806 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.187357 21124 raft_consensus.cc:3060] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.103500 24664 consensus_queue.cc:1048] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20251214 03:00:13.187692 21125 raft_consensus.cc:3060] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.188432 22013 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: []
W20251214 03:00:13.197489 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 137149us
W20251214 03:00:13.191907 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 84338.3us
I20251214 03:00:13.110128 24671 consensus_queue.cc:1048] T 2953f784a3f34b61a952ba296007ae31 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.085s
I20251214 03:00:13.204694 21248 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:13.200094 24696 log.cc:927] Time spent T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20251214 03:00:13.210688 24674 consensus_queue.cc:1048] T fb8de75de22740c889d4e6fe5fae1bc6 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20251214 03:00:13.203341 24572 consensus_queue.cc:1048] T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20251214 03:00:13.186510 24444 consensus_queue.cc:1048] T 5ce8bd94962e4bf48e01e9e747adba15 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20251214 03:00:13.168354 24616 raft_consensus.cc:515] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.189445 24666 raft_consensus.cc:2804] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:13.229965 21098 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 6e36caefa37d47a5968cffd9e270044f, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.233062 21248 leader_election.cc:304] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:13.235692 24695 log.cc:927] Time spent T 54e50911547a447cb80620e5c0ebb0b9 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20251214 03:00:13.237227 21098 leader_election.cc:304] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.238080 24572 raft_consensus.cc:2749] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251214 03:00:13.239367 21098 leader_election.cc:304] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:13.240274 21173 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.235558 24682 consensus_queue.cc:1048] T 639e38cdc0a74102bbcd86211abe87a0 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 03:00:13.240657 24572 raft_consensus.cc:2804] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:13.228878 24667 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:13.260021 24566 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:13.264698 21248 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.265240 21128 raft_consensus.cc:2468] T 143de581216646f79bece2b9f1259f64 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 10.
I20251214 03:00:13.265820 21248 leader_election.cc:304] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.264216 24566 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.270326 24566 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:13.268083 24601 raft_consensus.cc:2749] T 320811ca9d184da38a1197ebfdb7cd0f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:13.246033 24654 raft_consensus.cc:2749] T 01028fc01cbf4f4c88f7e36696d08d03 P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20251214 03:00:13.277038 24694 log.cc:927] Time spent T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
I20251214 03:00:13.281666 24436 raft_consensus.cc:493] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:13.292755 24601 raft_consensus.cc:2749] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:13.292906 24436 raft_consensus.cc:515] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.296233 24436 leader_election.cc:290] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.283777 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.297127 21133 raft_consensus.cc:3060] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:13.297742 24692 log.cc:927] Time spent T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20251214 03:00:13.299813 24698 consensus_queue.cc:1059] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251214 03:00:13.301295 24698 raft_consensus.cc:3055] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Stepping down as leader of term 7
I20251214 03:00:13.262624 24572 raft_consensus.cc:493] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
W20251214 03:00:13.286279 21247 leader_election.cc:336] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.657s (SENT)
I20251214 03:00:13.289408 21127 raft_consensus.cc:2468] T 6ae31f8b580840eb863e4d400367c1be P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 03:00:13.303006 24572 raft_consensus.cc:3060] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.285353 21248 leader_election.cc:304] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.301610 24698 raft_consensus.cc:740] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:13.244985 24678 raft_consensus.cc:2804] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 03:00:13.285966 21173 leader_election.cc:304] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.307657 21124 raft_consensus.cc:2468] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 10.
I20251214 03:00:13.279968 24566 raft_consensus.cc:515] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.263309 24667 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.312186 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:13.322131 21278 raft_consensus.cc:3060] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:13.229156 24666 raft_consensus.cc:493] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.314924 21276 raft_consensus.cc:1275] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251214 03:00:13.324822 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.297076 24613 leader_election.cc:290] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.332196 21279 raft_consensus.cc:1240] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20251214 03:00:13.335903 21247 leader_election.cc:336] T 6e37ffa0f0d64c3c8b5f77f26ff5f76f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.710s (SENT)
W20251214 03:00:13.339494 21984 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:13.361723 24699 log.cc:927] Time spent T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20251214 03:00:13.360752 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
W20251214 03:00:13.411711 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 197962us
I20251214 03:00:13.352532 21203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2953f784a3f34b61a952ba296007ae31" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:13.370853 24616 leader_election.cc:290] T 75e354ca56944ff1b7775a8309bca568 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:13.314417 24678 raft_consensus.cc:493] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
W20251214 03:00:13.361208 24659 consensus_meta.cc:287] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.001s
I20251214 03:00:13.375689 24698 consensus_queue.cc:260] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.419294 21133 raft_consensus.cc:2468] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 9.
I20251214 03:00:13.326043 24666 raft_consensus.cc:3060] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Advancing to term 9
W20251214 03:00:13.428804 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 143507us
I20251214 03:00:13.315799 24481 raft_consensus.cc:2804] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:13.430488 21125 raft_consensus.cc:2468] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 10.
I20251214 03:00:13.430821 21173 leader_election.cc:304] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.325127 24667 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:13.435288 21132 raft_consensus.cc:1275] T 5ce8bd94962e4bf48e01e9e747adba15 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 668ad4f30bd649c88a034bbab9f13b20: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20251214 03:00:13.425858 21986 leader_election.cc:304] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.439451 21248 leader_election.cc:304] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.419152 24576 raft_consensus.cc:2804] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:13.294064 24455 raft_consensus.cc:515] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: NON_VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: true } }
I20251214 03:00:13.461458 21248 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.431417 24678 raft_consensus.cc:3060] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Advancing to term 12
W20251214 03:00:13.464799 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 128927us
I20251214 03:00:13.466348 21248 leader_election.cc:304] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
W20251214 03:00:13.466645 21247 leader_election.cc:336] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.816s (SENT)
I20251214 03:00:13.469655 21248 leader_election.cc:304] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:13.460309 21030 catalog_manager.cc:5654] T 278f5a62c6a74101ae76cfc7325bcca0 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 4, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 4 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 03:00:13.460601 24576 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 668ad4f30bd649c88a034bbab9f13b20)
I20251214 03:00:13.472633 24576 raft_consensus.cc:3060] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:13.472352 21030 catalog_manager.cc:5654] T d55cde2542d544c5b99e6113b95910b8 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 7 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251214 03:00:13.468704 24677 raft_consensus.cc:2749] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 6e36caefa37d47a5968cffd9e270044f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251214 03:00:13.475247 24677 raft_consensus.cc:2804] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20251214 03:00:13.475576 24677 raft_consensus.cc:493] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 03:00:13.475898 24677 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:13.477869 24572 raft_consensus.cc:515] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.468811 24616 raft_consensus.cc:2749] T 2ce7181915c14c369bac4b962d34913a P 6e36caefa37d47a5968cffd9e270044f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251214 03:00:13.480270 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.480480 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.481329 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.473901 21030 catalog_manager.cc:5654] T a2f02493ca3c4039b61ed09e00355cc2 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 03:00:13.483686 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b832eb0688dc4d6c990d35c712ef6cc7" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.484632 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.482972 22011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:13.488461 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.491149 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e37ffa0f0d64c3c8b5f77f26ff5f76f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.491688 21132 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903"
dest_uuid: "721d604130924d939d291fe3eef8f6d1"
mode: GRACEFUL
new_leader_uuid: "668ad4f30bd649c88a034bbab9f13b20"
from {username='slave'} at 127.0.0.1:39374
I20251214 03:00:13.492197 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
W20251214 03:00:13.468955 24652 consensus_meta.cc:287] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.572s user 0.010s sys 0.000s
I20251214 03:00:13.490342 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96ca056dd7594a23a70918fc54c5eb2a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.489082 21030 catalog_manager.cc:5654] T 51f125bb79074d9fbb6c280fa03dc07e P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
W20251214 03:00:13.501607 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 175628us
I20251214 03:00:13.507105 21030 catalog_manager.cc:5654] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 6, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 6 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 03:00:13.508265 21173 leader_election.cc:304] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.509378 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.510027 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.510828 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.479951 24489 raft_consensus.cc:2749] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251214 03:00:13.508517 21030 catalog_manager.cc:5654] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251214 03:00:13.492354 21132 raft_consensus.cc:606] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Received request to transfer leadership to TS 668ad4f30bd649c88a034bbab9f13b20
I20251214 03:00:13.535399 21126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a54bd91b04344ee0b499fb117cc9fb91" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.482582 24681 raft_consensus.cc:2804] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:13.456849 24670 consensus_queue.cc:1059] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20251214 03:00:13.468286 24481 raft_consensus.cc:493] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.539213 21030 catalog_manager.cc:5654] T 3dcb97eb0c4e40bbabc681854107bb92 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 5 to 6, leader changed from 721d604130924d939d291fe3eef8f6d1 (127.19.131.65) to 6e36caefa37d47a5968cffd9e270044f (127.19.131.67). New cstate: current_term: 6 leader_uuid: "6e36caefa37d47a5968cffd9e270044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } health_report { overall_health: HEALTHY } } }
I20251214 03:00:13.543843 21173 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.549366 21030 catalog_manager.cc:5654] T ae2c1119874240b6af01af115f82ecd3 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 3 to 5, leader changed from 6e36caefa37d47a5968cffd9e270044f (127.19.131.67) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 5 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 03:00:13.551165 21125 raft_consensus.cc:1240] T 0ddd57244a03440195bc9142e5d4344a P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20251214 03:00:13.550210 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.553730 21124 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.553853 21985 leader_election.cc:304] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters:
W20251214 03:00:13.536587 24687 log.cc:927] Time spent T 5751534b61ed4568abd62057238d5fda P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251214 03:00:13.556196 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.557082 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.557601 21130 raft_consensus.cc:3060] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:13.576136 21129 raft_consensus.cc:2468] T e18f39bed60f44f5b8c25ede5d8bba8f P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 6.
I20251214 03:00:13.579936 21130 raft_consensus.cc:2468] T 2734ae274c16411d93b2c95d7fb3e00c P 721d604130924d939d291fe3eef8f6d1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 12.
I20251214 03:00:13.570878 22018 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b1dc02b9fd94ebd9df45c7602bc4cbd" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "8de0cf3ca4bf4083bc494a9af0e49342"
I20251214 03:00:13.587090 22011 raft_consensus.cc:1275] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 8de0cf3ca4bf4083bc494a9af0e49342 [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20251214 03:00:13.608732 21278 raft_consensus.cc:2468] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 8.
W20251214 03:00:13.560637 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 94037.6us
W20251214 03:00:13.613957 21247 leader_election.cc:336] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.867s (SENT)
I20251214 03:00:13.558907 21030 catalog_manager.cc:5654] T 75c047f1d6d94237afc050db2241d659 P 6e36caefa37d47a5968cffd9e270044f reported cstate change: term changed from 6 to 7, leader changed from 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66) to 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68). New cstate: current_term: 7 leader_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } } }
I20251214 03:00:13.539237 24681 raft_consensus.cc:493] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.538516 24566 leader_election.cc:290] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 6 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.619387 21125 raft_consensus.cc:1240] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 9. Current term is 10. Ops: []
I20251214 03:00:13.621241 21247 leader_election.cc:304] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.585407 21126 raft_consensus.cc:3055] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Stepping down as leader of term 6
I20251214 03:00:13.618240 24681 raft_consensus.cc:3060] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:13.551059 24444 raft_consensus.cc:3060] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.625943 22012 raft_consensus.cc:1240] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 LEADER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:13.628101 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.552450 24481 raft_consensus.cc:3060] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:13.626320 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 82561.2us
I20251214 03:00:13.628309 21126 raft_consensus.cc:740] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:13.630515 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.632391 21208 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251214 03:00:13.559676 24698 raft_consensus.cc:3060] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:13.564637 24454 raft_consensus.cc:493] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:13.631501 21173 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
W20251214 03:00:13.631578 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 161966us
I20251214 03:00:13.560546 24613 raft_consensus.cc:2804] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 FOLLOWER]: Leader election won for term 9
I20251214 03:00:13.560653 24572 leader_election.cc:290] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 10 election: Requested vote from peers 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525), 8de0cf3ca4bf4083bc494a9af0e49342 (127.19.131.68:33395)
I20251214 03:00:13.561846 24455 leader_election.cc:290] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 10 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.639693 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b5a42ee941246bf8d0050ea0dcfb9fc" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.643217 21248 leader_election.cc:304] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.646663 21248 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:13.626477 24538 raft_consensus.cc:2804] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election won for term 7
I20251214 03:00:13.644445 21130 raft_consensus.cc:2468] T 57d922c88eca4233b3dd4d314fac9e4a P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:13.584897 24533 raft_consensus.cc:515] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.648206 21248 leader_election.cc:304] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:13.653810 21248 leader_election.cc:304] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters:
I20251214 03:00:13.655913 21130 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20251214 03:00:13.564750 24677 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:13.656942 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 164631us
I20251214 03:00:13.581739 24666 raft_consensus.cc:515] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.661865 21173 leader_election.cc:304] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.640623 24616 raft_consensus.cc:2749] T 96ca056dd7594a23a70918fc54c5eb2a P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251214 03:00:13.665285 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 51362.7us
I20251214 03:00:13.650929 24489 raft_consensus.cc:2749] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:13.653043 24566 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251214 03:00:13.654174 21126 consensus_queue.cc:260] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.676038 21127 raft_consensus.cc:2368] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7.
I20251214 03:00:13.650767 24576 raft_consensus.cc:515] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
W20251214 03:00:13.677254 24704 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 721d604130924d939d291fe3eef8f6d1: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
I20251214 03:00:13.652930 24678 raft_consensus.cc:515] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.652926 24670 raft_consensus.cc:2804] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 03:00:13.687134 21126 raft_consensus.cc:3060] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:13.676190 21247 leader_election.cc:336] T a54bd91b04344ee0b499fb117cc9fb91 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855): Timed out: RequestConsensusVote RPC to 127.19.131.65:38855 timed out after 1.987s (SENT)
I20251214 03:00:13.659832 24538 raft_consensus.cc:697] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 LEADER]: Becoming Leader. State: Replica: 668ad4f30bd649c88a034bbab9f13b20, State: Running, Role: LEADER
I20251214 03:00:13.645210 24436 consensus_queue.cc:1059] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20251214 03:00:13.689256 21202 raft_consensus.cc:1240] T 8cd37022763049b2829947b4a33d6aea P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:13.677840 24566 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.667304 24613 raft_consensus.cc:697] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:13.647542 24454 raft_consensus.cc:515] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.695269 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.701931 21130 raft_consensus.cc:3060] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:13.701627 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8ab1ed25694443cb4efebfc35e5205e" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:13.706697 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 149250us
I20251214 03:00:13.694177 24616 raft_consensus.cc:2804] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251214 03:00:13.735360 21247 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 65930.4us
I20251214 03:00:13.701234 24666 leader_election.cc:290] T 275e8242241b4fd386a7c5c738a2c903 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 9 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.692075 24677 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.736829 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.692267 24670 raft_consensus.cc:493] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.738530 21128 raft_consensus.cc:1240] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20251214 03:00:13.692124 24678 leader_election.cc:290] T 5bdfefe7cfb64f5cba716a427e83fc9f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.703024 24566 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.690706 24515 consensus_queue.cc:1048] T 8cd37022763049b2829947b4a33d6aea P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20251214 03:00:13.685708 24690 raft_consensus.cc:515] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.755780 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.776793 21125 raft_consensus.cc:3060] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.777803 24690 leader_election.cc:290] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 12 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
W20251214 03:00:13.755102 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 101347us
I20251214 03:00:13.757648 24538 consensus_queue.cc:237] T d5a50d11dd114861bc3ebe1138e3d999 P 668ad4f30bd649c88a034bbab9f13b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.738348 24616 raft_consensus.cc:493] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.761699 21130 raft_consensus.cc:2468] T 8c3be94c517348a08049e54abf8fd780 P 721d604130924d939d291fe3eef8f6d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 6.
I20251214 03:00:13.762152 24396 raft_consensus.cc:2749] T 2953f784a3f34b61a952ba296007ae31 P 8de0cf3ca4bf4083bc494a9af0e49342 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20251214 03:00:13.762466 24713 log.cc:927] Time spent T f0e59ca14c1548a686e85b1a9fcf9288 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20251214 03:00:13.768744 24660 raft_consensus.cc:993] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1: : Instructing follower 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 03:00:13.773252 24533 leader_election.cc:290] T 3dcb97eb0c4e40bbabc681854107bb92 P 721d604130924d939d291fe3eef8f6d1 [CANDIDATE]: Term 7 election: Requested vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.811323 21199 raft_consensus.cc:1240] T b832eb0688dc4d6c990d35c712ef6cc7 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 5. Current term is 6. Ops: []
I20251214 03:00:13.774948 24264 raft_consensus.cc:2804] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251214 03:00:13.756031 24670 raft_consensus.cc:3060] T b98a871e4a86480da87fed8511a68575 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Advancing to term 7
I20251214 03:00:13.779186 21248 leader_election.cc:304] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.710351 24489 raft_consensus.cc:2804] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 03:00:13.778517 21208 raft_consensus.cc:1240] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 11. Current term is 12. Ops: []
I20251214 03:00:13.774596 24576 leader_election.cc:290] T b8b9d49c54144b37a99555815a60e1e4 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.753350 21985 leader_election.cc:304] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 721d604130924d939d291fe3eef8f6d1, 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20
W20251214 03:00:13.827415 21985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.65:38855, user_credentials={real_user=slave}} blocked reactor thread for 74225.4us
W20251214 03:00:13.756121 24567 consensus_meta.cc:287] T 4ba4da4df7bc466aa9cc42fa304335d8 P 721d604130924d939d291fe3eef8f6d1: Time spent flushing consensus metadata: real 0.615s user 0.005s sys 0.009s
I20251214 03:00:13.798708 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.798962 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639e38cdc0a74102bbcd86211abe87a0" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.799839 21129 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20251214 03:00:13.785745 24616 raft_consensus.cc:3060] T 2ca27c31805c4e55898fe4f60468ffa6 P 6e36caefa37d47a5968cffd9e270044f [term 9 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.803241 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:13.806668 21173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.67:34525, user_credentials={real_user=slave}} blocked reactor thread for 151752us
I20251214 03:00:13.807356 24454 leader_election.cc:290] T d5a50d11dd114861bc3ebe1138e3d999 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.807953 21125 raft_consensus.cc:2468] T 0c25ad2d7ede4ddeb75d46946114529b P 721d604130924d939d291fe3eef8f6d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 03:00:13.801632 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
W20251214 03:00:13.774770 24709 log.cc:927] Time spent T 2953f784a3f34b61a952ba296007ae31 P 668ad4f30bd649c88a034bbab9f13b20: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20251214 03:00:13.812059 21204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3dcb97eb0c4e40bbabc681854107bb92" candidate_uuid: "721d604130924d939d291fe3eef8f6d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.840673 21204 raft_consensus.cc:3060] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 6 FOLLOWER]: Advancing to term 7
W20251214 03:00:13.813848 21984 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251214 03:00:13.823148 21248 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 44041.7us
W20251214 03:00:13.822129 21986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.66:44049, user_credentials={real_user=slave}} blocked reactor thread for 164486us
I20251214 03:00:13.826349 21126 raft_consensus.cc:1275] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: Refusing update from remote peer 8de0cf3ca4bf4083bc494a9af0e49342: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20251214 03:00:13.842252 21126 pending_rounds.cc:85] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Aborting all ops after (but not including) 6
I20251214 03:00:13.842485 21126 pending_rounds.cc:107] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20251214 03:00:13.842723 21126 raft_consensus.cc:2889] T 8cd37022763049b2829947b4a33d6aea P 721d604130924d939d291fe3eef8f6d1 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251214 03:00:13.831447 21171 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.19.131.68:33395, user_credentials={real_user=slave}} blocked reactor thread for 59223.8us
I20251214 03:00:13.813565 24601 raft_consensus.cc:3055] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Stepping down as leader of term 8
I20251214 03:00:13.838982 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75e354ca56944ff1b7775a8309bca568" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.839326 21125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "205cf8366aa44b27a349916da03cff46" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.841660 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "275e8242241b4fd386a7c5c738a2c903" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.846936 21202 raft_consensus.cc:1240] T 51f125bb79074d9fbb6c280fa03dc07e P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20251214 03:00:13.826341 24660 raft_consensus.cc:1081] T 275e8242241b4fd386a7c5c738a2c903 P 721d604130924d939d291fe3eef8f6d1 [term 8 LEADER]: Signalling peer 668ad4f30bd649c88a034bbab9f13b20 to start an election
I20251214 03:00:13.824772 24264 raft_consensus.cc:493] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 8de0cf3ca4bf4083bc494a9af0e49342)
I20251214 03:00:13.837391 24444 raft_consensus.cc:2749] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251214 03:00:13.856158 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2734ae274c16411d93b2c95d7fb3e00c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.826370 24489 raft_consensus.cc:493] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.866051 21278 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ba6dea370cb421894536a9154735510" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.866400 24489 raft_consensus.cc:3060] T b832eb0688dc4d6c990d35c712ef6cc7 P 6e36caefa37d47a5968cffd9e270044f [term 5 FOLLOWER]: Advancing to term 6
I20251214 03:00:13.826233 24677 raft_consensus.cc:2749] T 4b611d0c068c451bb4eeb7eda4823036 P 6e36caefa37d47a5968cffd9e270044f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251214 03:00:13.858083 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e91314b7fb3c4dbd8db2eebcf16d820f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20" is_pre_election: true
I20251214 03:00:13.859853 24361 consensus_queue.cc:1048] T 6b5a42ee941246bf8d0050ea0dcfb9fc P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251214 03:00:13.853199 21173 leader_election.cc:304] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.833389 24456 raft_consensus.cc:2804] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 FOLLOWER]: Leader election won for term 6
I20251214 03:00:13.864514 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5751534b61ed4568abd62057238d5fda" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.873157 21274 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.859306 21985 leader_election.cc:304] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 6e36caefa37d47a5968cffd9e270044f, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.840370 24613 consensus_queue.cc:237] T 0ddd57244a03440195bc9142e5d4344a P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } } peers { permanent_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" member_type: VOTER last_known_addr { host: "127.19.131.68" port: 33395 } attrs { promote: false } }
I20251214 03:00:13.866649 21278 raft_consensus.cc:2468] T 0ba6dea370cb421894536a9154735510 P 6e36caefa37d47a5968cffd9e270044f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 2.
I20251214 03:00:13.869498 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.873363 21282 raft_consensus.cc:2468] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 8.
I20251214 03:00:13.873853 21274 raft_consensus.cc:2393] T fb8de75de22740c889d4e6fe5fae1bc6 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 in current term 7: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:13.877282 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2f02493ca3c4039b61ed09e00355cc2" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:13.878026 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.878415 21279 raft_consensus.cc:2368] T 54e50911547a447cb80620e5c0ebb0b9 P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668ad4f30bd649c88a034bbab9f13b20 for earlier term 9. Current term is 10.
I20251214 03:00:13.826300 24681 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.883004 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.851006 24601 raft_consensus.cc:740] T 0ddd57244a03440195bc9142e5d4344a P 6e36caefa37d47a5968cffd9e270044f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e36caefa37d47a5968cffd9e270044f, State: Running, Role: LEADER
I20251214 03:00:13.864143 24264 raft_consensus.cc:3060] T 320811ca9d184da38a1197ebfdb7cd0f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Advancing to term 8
I20251214 03:00:13.891731 24681 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.882248 21173 leader_election.cc:304] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.882884 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57d922c88eca4233b3dd4d314fac9e4a" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.892751 21201 raft_consensus.cc:2368] T 57d922c88eca4233b3dd4d314fac9e4a P 668ad4f30bd649c88a034bbab9f13b20 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f for earlier term 8. Current term is 9.
I20251214 03:00:13.893527 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3be94c517348a08049e54abf8fd780" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.878865 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51f125bb79074d9fbb6c280fa03dc07e" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:13.882015 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "434bbcb8330a4ba680c1346b33460471" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.894511 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e50911547a447cb80620e5c0ebb0b9" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.895099 21201 raft_consensus.cc:3060] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Advancing to term 10
I20251214 03:00:13.891317 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "668ad4f30bd649c88a034bbab9f13b20" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.889182 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55cde2542d544c5b99e6113b95910b8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.891301 21203 raft_consensus.cc:1240] T e0e04af2e6054ecabda3186cef55253f P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:13.895423 21202 raft_consensus.cc:2468] T e91314b7fb3c4dbd8db2eebcf16d820f P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 03:00:13.894938 21275 raft_consensus.cc:2468] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668ad4f30bd649c88a034bbab9f13b20 in term 7.
I20251214 03:00:13.901373 21281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.901575 21273 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5a50d11dd114861bc3ebe1138e3d999" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.902040 21281 raft_consensus.cc:2368] T 0c25ad2d7ede4ddeb75d46946114529b P 6e36caefa37d47a5968cffd9e270044f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 for earlier term 9. Current term is 10.
I20251214 03:00:13.902176 21273 raft_consensus.cc:2468] T d5a50d11dd114861bc3ebe1138e3d999 P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 7.
I20251214 03:00:13.906545 21279 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.909287 21282 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ba4da4df7bc466aa9cc42fa304335d8" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.888025 24436 raft_consensus.cc:2749] T 2b9c9a9914df4ca0aa72659e67ec2c4c P 8de0cf3ca4bf4083bc494a9af0e49342 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 03:00:13.915628 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b9c9a9914df4ca0aa72659e67ec2c4c" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.914811 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b98a871e4a86480da87fed8511a68575" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.913534 21282 raft_consensus.cc:2468] T 4ba4da4df7bc466aa9cc42fa304335d8 P 6e36caefa37d47a5968cffd9e270044f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de0cf3ca4bf4083bc494a9af0e49342 in term 4.
I20251214 03:00:13.876024 24720 consensus_queue.cc:1059] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20251214 03:00:13.917913 21275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3c57218e1ae40dfbdcec8dd228432ca" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f" is_pre_election: true
I20251214 03:00:13.909071 24456 raft_consensus.cc:697] T e18f39bed60f44f5b8c25ede5d8bba8f P 8de0cf3ca4bf4083bc494a9af0e49342 [term 6 LEADER]: Becoming Leader. State: Replica: 8de0cf3ca4bf4083bc494a9af0e49342, State: Running, Role: LEADER
I20251214 03:00:13.921808 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb8de75de22740c889d4e6fe5fae1bc6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.914878 24572 consensus_queue.cc:1059] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
W20251214 03:00:13.922984 24715 log.cc:927] Time spent T f692dd0fa4b64450b9808a14da2b1c29 P 6e36caefa37d47a5968cffd9e270044f: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20251214 03:00:13.923380 21277 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ddd57244a03440195bc9142e5d4344a" candidate_uuid: "8de0cf3ca4bf4083bc494a9af0e49342" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e36caefa37d47a5968cffd9e270044f"
I20251214 03:00:13.926647 24667 raft_consensus.cc:515] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.939563 21275 raft_consensus.cc:2393] T a3c57218e1ae40dfbdcec8dd228432ca P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8de0cf3ca4bf4083bc494a9af0e49342 in current term 8: Already voted for candidate 6e36caefa37d47a5968cffd9e270044f in this term.
I20251214 03:00:13.940595 24667 leader_election.cc:290] T e0e04af2e6054ecabda3186cef55253f P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 668ad4f30bd649c88a034bbab9f13b20 (127.19.131.66:44049)
I20251214 03:00:13.873266 24481 raft_consensus.cc:515] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.942094 21199 raft_consensus.cc:1240] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Rejecting Update request from peer 721d604130924d939d291fe3eef8f6d1 for earlier term 6. Current term is 7. Ops: []
I20251214 03:00:13.923931 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8b9d49c54144b37a99555815a60e1e4" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"
I20251214 03:00:13.943516 21131 raft_consensus.cc:2393] T b8b9d49c54144b37a99555815a60e1e4 P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 8: Already voted for candidate 721d604130924d939d291fe3eef8f6d1 in this term.
I20251214 03:00:13.944094 24572 raft_consensus.cc:3055] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Stepping down as leader of term 11
I20251214 03:00:13.944433 24572 raft_consensus.cc:740] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:13.945194 24572 consensus_queue.cc:260] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "721d604130924d939d291fe3eef8f6d1" member_type: VOTER last_known_addr { host: "127.19.131.65" port: 38855 } } peers { permanent_uuid: "668ad4f30bd649c88a034bbab9f13b20" member_type: VOTER last_known_addr { host: "127.19.131.66" port: 44049 } } peers { permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 } }
I20251214 03:00:13.924255 21208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01028fc01cbf4f4c88f7e36696d08d03" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.925105 21173 leader_election.cc:304] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.943380 21206 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c25ad2d7ede4ddeb75d46946114529b" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.923956 24723 raft_consensus.cc:493] T b8b9d49c54144b37a99555815a60e1e4 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 03:00:13.942919 24481 leader_election.cc:290] T 639e38cdc0a74102bbcd86211abe87a0 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 election: Requested vote from peers 721d604130924d939d291fe3eef8f6d1 (127.19.131.65:38855), 6e36caefa37d47a5968cffd9e270044f (127.19.131.67:34525)
I20251214 03:00:13.951133 21206 raft_consensus.cc:2393] T 0c25ad2d7ede4ddeb75d46946114529b P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 10: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:13.953007 21133 raft_consensus.cc:2468] T a3c57218e1ae40dfbdcec8dd228432ca P 721d604130924d939d291fe3eef8f6d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 8.
I20251214 03:00:13.938355 21204 raft_consensus.cc:2468] T 3dcb97eb0c4e40bbabc681854107bb92 P 668ad4f30bd649c88a034bbab9f13b20 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721d604130924d939d291fe3eef8f6d1 in term 7.
I20251214 03:00:13.958492 24720 raft_consensus.cc:3055] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Stepping down as leader of term 5
I20251214 03:00:13.958920 24720 raft_consensus.cc:740] T b832eb0688dc4d6c990d35c712ef6cc7 P 721d604130924d939d291fe3eef8f6d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 721d604130924d939d291fe3eef8f6d1, State: Running, Role: LEADER
I20251214 03:00:13.959075 21201 raft_consensus.cc:2468] T 54e50911547a447cb80620e5c0ebb0b9 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e36caefa37d47a5968cffd9e270044f in term 10.
I20251214 03:00:13.960256 24681 raft_consensus.cc:2804] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 03:00:13.960697 24681 raft_consensus.cc:493] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.914825 24444 raft_consensus.cc:2804] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251214 03:00:13.960374 21173 leader_election.cc:304] T b98a871e4a86480da87fed8511a68575 P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.961067 24681 raft_consensus.cc:3060] T 755b818717824383985748e28244d7a1 P 668ad4f30bd649c88a034bbab9f13b20 [term 3 FOLLOWER]: Advancing to term 4
I20251214 03:00:13.966647 24444 raft_consensus.cc:493] T 205cf8366aa44b27a349916da03cff46 P 668ad4f30bd649c88a034bbab9f13b20 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 721d604130924d939d291fe3eef8f6d1)
I20251214 03:00:13.971342 24728 consensus_queue.cc:1059] T c8ab1ed25694443cb4efebfc35e5205e P 8de0cf3ca4bf4083bc494a9af0e49342 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e36caefa37d47a5968cffd9e270044f" member_type: VOTER last_known_addr { host: "127.19.131.67" port: 34525 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251214 03:00:13.974351 21173 leader_election.cc:304] T bf5cb5c0f4854e4a846c25eb5ce80a9a P 668ad4f30bd649c88a034bbab9f13b20 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f; no voters: 8de0cf3ca4bf4083bc494a9af0e49342
I20251214 03:00:13.974885 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "755b818717824383985748e28244d7a1" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1" is_pre_election: true
I20251214 03:00:13.975792 21202 raft_consensus.cc:1240] T a2f02493ca3c4039b61ed09e00355cc2 P 668ad4f30bd649c88a034bbab9f13b20 [term 10 FOLLOWER]: Rejecting Update request from peer 6e36caefa37d47a5968cffd9e270044f for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20251214 03:00:13.983073 21202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5bdfefe7cfb64f5cba716a427e83fc9f" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:13.985991 21247 leader_election.cc:304] T 434bbcb8330a4ba680c1346b33460471 P 6e36caefa37d47a5968cffd9e270044f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e36caefa37d47a5968cffd9e270044f; no voters: 668ad4f30bd649c88a034bbab9f13b20, 721d604130924d939d291fe3eef8f6d1
I20251214 03:00:13.991358 21276 raft_consensus.cc:1275] T 5751534b61ed4568abd62057238d5fda P 6e36caefa37d47a5968cffd9e270044f [term 8 FOLLOWER]: Refusing update from remote peer 721d604130924d939d291fe3eef8f6d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251214 03:00:13.979004 24572 raft_consensus.cc:3060] T 9695e153d2f54b008f8c7201463cfbc8 P 721d604130924d939d291fe3eef8f6d1 [term 11 FOLLOWER]: Advancing to term 12
I20251214 03:00:13.959208 24561 raft_consensus.cc:3055] T 1b1dc02b9fd94ebd9df45c7602bc4cbd P 8de0cf3ca4bf4083bc494a9af0e49342 [term 9 LEADER]: Stepping down as leader of term 9
I20251214 03:00:13.996245 21985 leader_election.cc:304] T d55cde2542d544c5b99e6113b95910b8 P 8de0cf3ca4bf4083bc494a9af0e49342 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8de0cf3ca4bf4083bc494a9af0e49342; no voters: 668ad4f30bd649c88a034bbab9f13b20, 6e36caefa37d47a5968cffd9e270044f
I20251214 03:00:14.003427 21201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9695e153d2f54b008f8c7201463cfbc8" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "668ad4f30bd649c88a034bbab9f13b20"
I20251214 03:00:14.004390 21201 raft_consensus.cc:2393] T 9695e153d2f54b008f8c7201463cfbc8 P 668ad4f30bd649c88a034bbab9f13b20 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e36caefa37d47a5968cffd9e270044f in current term 12: Already voted for candidate 668ad4f30bd649c88a034bbab9f13b20 in this term.
I20251214 03:00:13.994828 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca27c31805c4e55898fe4f60468ffa6" candidate_uuid: "6e36caefa37d47a5968cffd9e270044f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "721d604130924d939d291fe3eef8f6d1"